Skip to content

fix(api): resolve aliases on single-index endpoints; persist aliases across restart#30

Open
Vinz2168 wants to merge 1 commit into
xerj-org:mainfrom
Vinz2168:fix/opensearch-alias-persistence
Open

fix(api): resolve aliases on single-index endpoints; persist aliases across restart#30
Vinz2168 wants to merge 1 commit into
xerj-org:mainfrom
Vinz2168:fix/opensearch-alias-persistence

Conversation

@Vinz2168

Copy link
Copy Markdown

Summary

Follow-up to #28/#29. Root-caused a real, reproducible OpenSearch Dashboards bug: a fresh OSD container, pointed at an xerj instance that already has a fully migrated .kibana index, gets stuck indefinitely on:

Another OpenSearch Dashboards instance appears to be migrating the index.
Waiting for that migration to complete.

— even though there is only ever one real instance. This blocks OSD from starting at all on any second/subsequent run against existing data.

Root cause — two compounding bugs, found via a raw byte-level TCP relay between a real OSD 3.6.0 container and xerj (same technique used for the earlier _cat/templates fix)

1. Single-index endpoints never resolved an alias to its underlying real index. .kibana is always an alias, never a bare index — OSD/Kibana create .kibana_1 then POST _aliases to point .kibana at it. So:

  • GET /.kibana → 404 index_not_found_exception, even though .kibana was a registered alias.
  • GET /.kibana/_mapping → 200, but missing mappings._meta.migrationMappingPropertyHashes entirely — silently falling back to a schema-derived reconstruction instead of the actually-stored mapping blob.

_meta.migrationMappingPropertyHashes is exactly the field OSD's savedobjects migrator reads to decide "already migrated, adopt it" vs "someone else is migrating, wait." Both gaps meant it could never see a completed migration. Fixed by resolving aliases in get_index_inner's inline selector and the shared resolve_index_selector (used by get_mapping/get_settings/etc.) — matching real ES/OpenSearch, which accept a real index name or an alias name interchangeably on every single-index endpoint.

2. The actual root cause: engine.aliases was never persisted to disk. Index data survives an xerj restart (confirmed working all night); the alias pointing at it didn't. So even after (1) was fixed and an alias was manually recreated via _aliases, a real restart of xerj wiped it again, and the next OSD container hit the identical stuck state. Added aliases.json persistence — atomic temp+rename, the same pattern already used for es_mapping.json (per-index) and api_keys.json (global): flushed on every mutation (add_alias, remove_alias, delete_index's cleanup, data-stream rollover/delete), loaded once at Engine::new() before the server accepts requests.

Verified end-to-end

Real OpenSearch Dashboards 3.6.0 container, repeated across three separate xerj restarts in a row: no manual alias recreation, no lock warning, straight to Server running / status.overall.state: "green" every single time. Previously this needed manual _aliases API intervention after every restart, indefinitely.

Test plan

  • cargo build --release -p xerj-engine -p xerj-api -p xerj-server
  • cargo fmt --check / cargo clippy --no-deps — clean
  • curl: GET /.kibana and GET /.kibana/_mapping correctly resolve through the alias, _meta intact
  • curl: create index + alias, restart xerj, alias survives (_cat/aliases, GET /.kibana both correct) with zero manual intervention
  • Full ES-compat YAML conformance suite: 1360 passed, 0 failed, 3 skipped — no regressions
  • Real OpenSearch Dashboards 3.6.0 container: 3 consecutive xerj restarts, each followed by a fresh OSD container — clean startup every time, no manual fixup

🤖 Generated with Claude Code

…across restart

Root-caused a real OpenSearch Dashboards bug where a FRESH OSD
container, pointed at an xerj instance that already has a fully
migrated `.kibana` index, gets stuck forever on:

  "Another OpenSearch Dashboards instance appears to be migrating
  the index. Waiting for that migration to complete."

...even though there is only ever one real instance. Traced via a
raw byte-level TCP relay between a real OSD 3.6.0 container and
xerj (same technique as the earlier _cat/templates fix) to two
distinct, compounding bugs:

1. GET /{alias} and GET /{alias}/_mapping never resolved an alias to
   its underlying real index. `.kibana` is ALWAYS an alias, never a
   bare index (OSD/Kibana create `.kibana_1` then alias `.kibana` to
   it) -- so `GET /.kibana` 404'd outright, and `GET /.kibana/_mapping`
   returned a schema-derived reconstruction missing
   `mappings._meta.migrationMappingPropertyHashes`, the exact field
   OSD's savedobjects migrator reads to decide "already migrated,
   adopt it" vs "someone else is migrating, wait". Fixed by resolving
   aliases in `get_index_inner`'s inline selector and in the shared
   `resolve_index_selector` (used by get_mapping/get_settings/etc.),
   mirroring how ES/OpenSearch accept a real index name OR an alias
   name interchangeably on every single-index endpoint.

2. The actual root cause: `engine.aliases` was NEVER persisted to
   disk. Index DATA survives a restart (confirmed working all
   night), but the alias pointing at it didn't -- so even after (1)
   was fixed and an alias was manually recreated, a REAL restart of
   xerj still wiped it, and the very next OSD container hit the
   identical stuck state again. Added `aliases.json` persistence
   (atomic temp+rename, same pattern already used for
   `es_mapping.json` per-index and `api_keys.json` globally):
   flushed on every mutation (`add_alias`, `remove_alias`,
   `delete_index`'s alias cleanup, data-stream rollover/delete), and
   loaded once at `Engine::new()` startup, before the server accepts
   requests.

Verified end-to-end against a real OpenSearch Dashboards 3.6.0
container, repeated across THREE separate xerj restarts in a row: no
manual alias recreation, no lock warning, straight to "Server
running" / `status.overall.state: "green"` every time -- previously
reproduced needing manual `_aliases` intervention after every single
restart.

Full ES-compat YAML conformance suite: 1360 passed, 0 failed, 3
skipped -- no regressions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016XJaZygeuRfZfUg2B8tPKU
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant