Skip to content

[8.19] [ML] Extend reindexed AD results template and generalize heal beyond job_id (#153755) - #159079

Merged
valeriy42 merged 3 commits into
elastic:8.19from
valeriy42:backport/8.19/pr-153755
Sep 16, 2026
Merged

valeriy42 merged 3 commits into
elastic:8.19from
valeriy42:backport/8.19/pr-153755

Conversation

@valeriy42

Copy link
Copy Markdown
Contributor

Backport

This will backport the following commits from main to 8.19:

Questions ?

Please refer to the Backport tool documentation

…job_id (elastic#153755)

## Summary

Fixes a class of ML anomaly-detection failures on clusters upgraded through 8.18/8.19 where `.reindexed-*-ml-anomalies-*` results indices were created without the managed AD template mappings. Jobs can enter `failed` with `failed to update results mapping` when opening, or silently write to indices whose mappings conflict with the current template.

This PR combines two complementary fixes:
1. **Preventive** — widen the AD results composable template pattern from `.reindexed-v7-ml-anomalies-*` to `.reindexed-*-ml-anomalies-*` (matching the heal wildcard already in `MlAnomaliesIndexUpdate`), and bump `ML_INDEX_TEMPLATE_VERSION` to `10000003` so existing clusters reinstall the template (elastic#138097 lesson from elastic#135270).
2. **Corrective** — extend the runtime heal from elastic#147688 so it no longer gates only on `job_id: keyword`; indices with correct `job_id` but wrong `anomaly_score_explanation.*` types (e.g. `by_field_relative_rarity: float` instead of `double`) are also healed by moving aliases to a template-backed target index.

## Changes

Template (prevent future corruption)

- `results_index_template.json`: `.reindexed-v7-ml-anomalies-*` → `.reindexed-*-ml-anomalies-*`
- `MlIndexTemplateRegistry`: version bump to `10000003` with javadoc entry
- Cross-reference javadoc on `REINDEXED_ANOMALIES_PATTERN` ↔ template resource
- `MlIndexTemplateRegistryTests.testResultsTemplate()` — new regression pinning the pattern list
- `MlMappingsUpgradeIT` — updated version constant and asserted patterns
- `machine-learning-settings.md` — heal setting description aligned to the wildcard

Heal (repair already-broken clusters)

- `MlIndexAndAlias.hasFieldTypedAs(IndexMetadata, List<String> fieldPath, …)` — nested-path lookup for `anomaly_score_explanation.*`
- `MlAnomaliesIndexUpdate.isIndexMappingHealthy()` — requires `job_id: keyword` **and** all four template `double` fields under `anomaly_score_explanation` (`lower_confidence_bound`, `typical_value`, `upper_confidence_bound`, `by_field_relative_rarity`)
- Gate applied in both heal detection (`healOneBadIndex`) and target reuse (`resolveOrCreateTargetIndex`)
- Advisory notification text generalized beyond `job_id`; affected-version list updated
- `MlAnomaliesIndexUpdateTests.testHealReindexedV7_HealWhenJobIdKeywordButAnomalyScoreExplanationFloat` — regression for the platform-logging failure mode

## Notes

- The heal step is gated by `xpack.ml.anomalies.heal_reindexed_v7.enabled` (default `true`, dynamically updatable).
- Heal moves aliases only; it does not reindex historical results. Stranded data recovery still requires the `_reindex` procedure documented in elastic#147686 notifications.
- Template version bump is required for the preventive fix to reach already-provisioned clusters (same deploy mechanism as elastic#138097).
- Backport targets: `8.19`, `9.3`, `9.4` (branches already carry job_id-only heal from elastic#153399/elastic#153397/elastic#153396).

Complements and extends the runtime heal in elastic#147688.

(cherry picked from commit 6161f77)

# Conflicts:
#	docs/reference/elasticsearch/configuration-reference/machine-learning-settings.md
#	x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/MlIndexTemplateRegistry.java
#	x-pack/qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/MlMappingsUpgradeIT.java
@valeriy42 valeriy42 added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Sep 16, 2026
@valeriy42
valeriy42 merged commit 76353a1 into elastic:8.19 Sep 16, 2026
24 checks passed
@valeriy42
valeriy42 deleted the backport/8.19/pr-153755 branch September 16, 2026 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) backport >bug :ml Machine learning Team:ML Meta label for the ML team v8.19.22

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants