[ML] Bump anomalies index template version - #138097

* [ML] Bump anomalies index template version (#138097)

* fix compile

…job_id (#153755)

## Summary

Fixes a class of ML anomaly-detection failures on clusters upgraded through 8.18/8.19 where `.reindexed-*-ml-anomalies-*` results indices were created without the managed AD template mappings. Jobs can enter `failed` with `failed to update results mapping` when opening, or silently write to indices whose mappings conflict with the current template.

This PR combines two complementary fixes:
1. **Preventive** — widen the AD results composable template pattern from `.reindexed-v7-ml-anomalies-*` to `.reindexed-*-ml-anomalies-*` (matching the heal wildcard already in `MlAnomaliesIndexUpdate`), and bump `ML_INDEX_TEMPLATE_VERSION` to `10000003` so existing clusters reinstall the template (#138097 lesson from #135270).
2. **Corrective** — extend the runtime heal from #147688 so it no longer gates only on `job_id: keyword`; indices with correct `job_id` but wrong `anomaly_score_explanation.*` types (e.g. `by_field_relative_rarity: float` instead of `double`) are also healed by moving aliases to a template-backed target index.

## Changes

Template (prevent future corruption)

- `results_index_template.json`: `.reindexed-v7-ml-anomalies-*` → `.reindexed-*-ml-anomalies-*`
- `MlIndexTemplateRegistry`: version bump to `10000003` with javadoc entry
- Cross-reference javadoc on `REINDEXED_ANOMALIES_PATTERN` ↔ template resource
- `MlIndexTemplateRegistryTests.testResultsTemplate()` — new regression pinning the pattern list
- `MlMappingsUpgradeIT` — updated version constant and asserted patterns
- `machine-learning-settings.md` — heal setting description aligned to the wildcard

Heal (repair already-broken clusters)

- `MlIndexAndAlias.hasFieldTypedAs(IndexMetadata, List<String> fieldPath, …)` — nested-path lookup for `anomaly_score_explanation.*`
- `MlAnomaliesIndexUpdate.isIndexMappingHealthy()` — requires `job_id: keyword` **and** all four template `double` fields under `anomaly_score_explanation` (`lower_confidence_bound`, `typical_value`, `upper_confidence_bound`, `by_field_relative_rarity`)
- Gate applied in both heal detection (`healOneBadIndex`) and target reuse (`resolveOrCreateTargetIndex`)
- Advisory notification text generalized beyond `job_id`; affected-version list updated
- `MlAnomaliesIndexUpdateTests.testHealReindexedV7_HealWhenJobIdKeywordButAnomalyScoreExplanationFloat` — regression for the platform-logging failure mode

## Notes

- The heal step is gated by `xpack.ml.anomalies.heal_reindexed_v7.enabled` (default `true`, dynamically updatable).
- Heal moves aliases only; it does not reindex historical results. Stranded data recovery still requires the `_reindex` procedure documented in #147686 notifications.
- Template version bump is required for the preventive fix to reach already-provisioned clusters (same deploy mechanism as #138097).
- Backport targets: `8.19`, `9.3`, `9.4` (branches already carry job_id-only heal from #153399/#153397/#153396).

Complements and extends the runtime heal in #147688.

…job_id (#153755) (#153931)

## Summary

Fixes a class of ML anomaly-detection failures on clusters upgraded through 8.18/8.19 where `.reindexed-*-ml-anomalies-*` results indices were created without the managed AD template mappings. Jobs can enter `failed` with `failed to update results mapping` when opening, or silently write to indices whose mappings conflict with the current template.

This PR combines two complementary fixes:
1. **Preventive** — widen the AD results composable template pattern from `.reindexed-v7-ml-anomalies-*` to `.reindexed-*-ml-anomalies-*` (matching the heal wildcard already in `MlAnomaliesIndexUpdate`), and bump `ML_INDEX_TEMPLATE_VERSION` to `10000003` so existing clusters reinstall the template (#138097 lesson from #135270).
2. **Corrective** — extend the runtime heal from #147688 so it no longer gates only on `job_id: keyword`; indices with correct `job_id` but wrong `anomaly_score_explanation.*` types (e.g. `by_field_relative_rarity: float` instead of `double`) are also healed by moving aliases to a template-backed target index.

## Changes

Template (prevent future corruption)

- `results_index_template.json`: `.reindexed-v7-ml-anomalies-*` → `.reindexed-*-ml-anomalies-*`
- `MlIndexTemplateRegistry`: version bump to `10000003` with javadoc entry
- Cross-reference javadoc on `REINDEXED_ANOMALIES_PATTERN` ↔ template resource
- `MlIndexTemplateRegistryTests.testResultsTemplate()` — new regression pinning the pattern list
- `MlMappingsUpgradeIT` — updated version constant and asserted patterns
- `machine-learning-settings.md` — heal setting description aligned to the wildcard

Heal (repair already-broken clusters)

- `MlIndexAndAlias.hasFieldTypedAs(IndexMetadata, List<String> fieldPath, …)` — nested-path lookup for `anomaly_score_explanation.*`
- `MlAnomaliesIndexUpdate.isIndexMappingHealthy()` — requires `job_id: keyword` **and** all four template `double` fields under `anomaly_score_explanation` (`lower_confidence_bound`, `typical_value`, `upper_confidence_bound`, `by_field_relative_rarity`)
- Gate applied in both heal detection (`healOneBadIndex`) and target reuse (`resolveOrCreateTargetIndex`)
- Advisory notification text generalized beyond `job_id`; affected-version list updated
- `MlAnomaliesIndexUpdateTests.testHealReindexedV7_HealWhenJobIdKeywordButAnomalyScoreExplanationFloat` — regression for the platform-logging failure mode

## Notes

- The heal step is gated by `xpack.ml.anomalies.heal_reindexed_v7.enabled` (default `true`, dynamically updatable).
- Heal moves aliases only; it does not reindex historical results. Stranded data recovery still requires the `_reindex` procedure documented in #147686 notifications.
- Template version bump is required for the preventive fix to reach already-provisioned clusters (same deploy mechanism as #138097).
- Backport targets: `8.19`, `9.3`, `9.4` (branches already carry job_id-only heal from #153399/#153397/#153396).

Complements and extends the runtime heal in #147688.

…job_id (#153755) (#153932)

## Summary

Fixes a class of ML anomaly-detection failures on clusters upgraded through 8.18/8.19 where `.reindexed-*-ml-anomalies-*` results indices were created without the managed AD template mappings. Jobs can enter `failed` with `failed to update results mapping` when opening, or silently write to indices whose mappings conflict with the current template.

This PR combines two complementary fixes:
1. **Preventive** — widen the AD results composable template pattern from `.reindexed-v7-ml-anomalies-*` to `.reindexed-*-ml-anomalies-*` (matching the heal wildcard already in `MlAnomaliesIndexUpdate`), and bump `ML_INDEX_TEMPLATE_VERSION` to `10000003` so existing clusters reinstall the template (#138097 lesson from #135270).
2. **Corrective** — extend the runtime heal from #147688 so it no longer gates only on `job_id: keyword`; indices with correct `job_id` but wrong `anomaly_score_explanation.*` types (e.g. `by_field_relative_rarity: float` instead of `double`) are also healed by moving aliases to a template-backed target index.

## Changes

Template (prevent future corruption)

- `results_index_template.json`: `.reindexed-v7-ml-anomalies-*` → `.reindexed-*-ml-anomalies-*`
- `MlIndexTemplateRegistry`: version bump to `10000003` with javadoc entry
- Cross-reference javadoc on `REINDEXED_ANOMALIES_PATTERN` ↔ template resource
- `MlIndexTemplateRegistryTests.testResultsTemplate()` — new regression pinning the pattern list
- `MlMappingsUpgradeIT` — updated version constant and asserted patterns
- `machine-learning-settings.md` — heal setting description aligned to the wildcard

Heal (repair already-broken clusters)

- `MlIndexAndAlias.hasFieldTypedAs(IndexMetadata, List<String> fieldPath, …)` — nested-path lookup for `anomaly_score_explanation.*`
- `MlAnomaliesIndexUpdate.isIndexMappingHealthy()` — requires `job_id: keyword` **and** all four template `double` fields under `anomaly_score_explanation` (`lower_confidence_bound`, `typical_value`, `upper_confidence_bound`, `by_field_relative_rarity`)
- Gate applied in both heal detection (`healOneBadIndex`) and target reuse (`resolveOrCreateTargetIndex`)
- Advisory notification text generalized beyond `job_id`; affected-version list updated
- `MlAnomaliesIndexUpdateTests.testHealReindexedV7_HealWhenJobIdKeywordButAnomalyScoreExplanationFloat` — regression for the platform-logging failure mode

## Notes

- The heal step is gated by `xpack.ml.anomalies.heal_reindexed_v7.enabled` (default `true`, dynamically updatable).
- Heal moves aliases only; it does not reindex historical results. Stranded data recovery still requires the `_reindex` procedure documented in #147686 notifications.
- Template version bump is required for the preventive fix to reach already-provisioned clusters (same deploy mechanism as #138097).
- Backport targets: `8.19`, `9.3`, `9.4` (branches already carry job_id-only heal from #153399/#153397/#153396).

Complements and extends the runtime heal in #147688.

…job_id (#153755) (#153930)

## Summary

Fixes a class of ML anomaly-detection failures on clusters upgraded through 8.18/8.19 where `.reindexed-*-ml-anomalies-*` results indices were created without the managed AD template mappings. Jobs can enter `failed` with `failed to update results mapping` when opening, or silently write to indices whose mappings conflict with the current template.

This PR combines two complementary fixes:
1. **Preventive** — widen the AD results composable template pattern from `.reindexed-v7-ml-anomalies-*` to `.reindexed-*-ml-anomalies-*` (matching the heal wildcard already in `MlAnomaliesIndexUpdate`), and bump `ML_INDEX_TEMPLATE_VERSION` to `10000003` so existing clusters reinstall the template (#138097 lesson from #135270).
2. **Corrective** — extend the runtime heal from #147688 so it no longer gates only on `job_id: keyword`; indices with correct `job_id` but wrong `anomaly_score_explanation.*` types (e.g. `by_field_relative_rarity: float` instead of `double`) are also healed by moving aliases to a template-backed target index.

## Changes

Template (prevent future corruption)

- `results_index_template.json`: `.reindexed-v7-ml-anomalies-*` → `.reindexed-*-ml-anomalies-*`
- `MlIndexTemplateRegistry`: version bump to `10000003` with javadoc entry
- Cross-reference javadoc on `REINDEXED_ANOMALIES_PATTERN` ↔ template resource
- `MlIndexTemplateRegistryTests.testResultsTemplate()` — new regression pinning the pattern list
- `MlMappingsUpgradeIT` — updated version constant and asserted patterns
- `machine-learning-settings.md` — heal setting description aligned to the wildcard

Heal (repair already-broken clusters)

- `MlIndexAndAlias.hasFieldTypedAs(IndexMetadata, List<String> fieldPath, …)` — nested-path lookup for `anomaly_score_explanation.*`
- `MlAnomaliesIndexUpdate.isIndexMappingHealthy()` — requires `job_id: keyword` **and** all four template `double` fields under `anomaly_score_explanation` (`lower_confidence_bound`, `typical_value`, `upper_confidence_bound`, `by_field_relative_rarity`)
- Gate applied in both heal detection (`healOneBadIndex`) and target reuse (`resolveOrCreateTargetIndex`)
- Advisory notification text generalized beyond `job_id`; affected-version list updated
- `MlAnomaliesIndexUpdateTests.testHealReindexedV7_HealWhenJobIdKeywordButAnomalyScoreExplanationFloat` — regression for the platform-logging failure mode

## Notes

- The heal step is gated by `xpack.ml.anomalies.heal_reindexed_v7.enabled` (default `true`, dynamically updatable).
- Heal moves aliases only; it does not reindex historical results. Stranded data recovery still requires the `_reindex` procedure documented in #147686 notifications.
- Template version bump is required for the preventive fix to reach already-provisioned clusters (same deploy mechanism as #138097).
- Backport targets: `8.19`, `9.3`, `9.4` (branches already carry job_id-only heal from #153399/#153397/#153396).

Complements and extends the runtime heal in #147688.

…beyond job_id (#153755) (#159079)

* [ML] Extend reindexed AD results template and generalize heal beyond job_id (#153755)

## Summary

Fixes a class of ML anomaly-detection failures on clusters upgraded through 8.18/8.19 where `.reindexed-*-ml-anomalies-*` results indices were created without the managed AD template mappings. Jobs can enter `failed` with `failed to update results mapping` when opening, or silently write to indices whose mappings conflict with the current template.

This PR combines two complementary fixes:
1. **Preventive** — widen the AD results composable template pattern from `.reindexed-v7-ml-anomalies-*` to `.reindexed-*-ml-anomalies-*` (matching the heal wildcard already in `MlAnomaliesIndexUpdate`), and bump `ML_INDEX_TEMPLATE_VERSION` to `10000003` so existing clusters reinstall the template (#138097 lesson from #135270).
2. **Corrective** — extend the runtime heal from #147688 so it no longer gates only on `job_id: keyword`; indices with correct `job_id` but wrong `anomaly_score_explanation.*` types (e.g. `by_field_relative_rarity: float` instead of `double`) are also healed by moving aliases to a template-backed target index.

## Changes

Template (prevent future corruption)

- `results_index_template.json`: `.reindexed-v7-ml-anomalies-*` → `.reindexed-*-ml-anomalies-*`
- `MlIndexTemplateRegistry`: version bump to `10000003` with javadoc entry
- Cross-reference javadoc on `REINDEXED_ANOMALIES_PATTERN` ↔ template resource
- `MlIndexTemplateRegistryTests.testResultsTemplate()` — new regression pinning the pattern list
- `MlMappingsUpgradeIT` — updated version constant and asserted patterns
- `machine-learning-settings.md` — heal setting description aligned to the wildcard

Heal (repair already-broken clusters)

- `MlIndexAndAlias.hasFieldTypedAs(IndexMetadata, List<String> fieldPath, …)` — nested-path lookup for `anomaly_score_explanation.*`
- `MlAnomaliesIndexUpdate.isIndexMappingHealthy()` — requires `job_id: keyword` **and** all four template `double` fields under `anomaly_score_explanation` (`lower_confidence_bound`, `typical_value`, `upper_confidence_bound`, `by_field_relative_rarity`)
- Gate applied in both heal detection (`healOneBadIndex`) and target reuse (`resolveOrCreateTargetIndex`)
- Advisory notification text generalized beyond `job_id`; affected-version list updated
- `MlAnomaliesIndexUpdateTests.testHealReindexedV7_HealWhenJobIdKeywordButAnomalyScoreExplanationFloat` — regression for the platform-logging failure mode

## Notes

- The heal step is gated by `xpack.ml.anomalies.heal_reindexed_v7.enabled` (default `true`, dynamically updatable).
- Heal moves aliases only; it does not reindex historical results. Stranded data recovery still requires the `_reindex` procedure documented in #147686 notifications.
- Template version bump is required for the preventive fix to reach already-provisioned clusters (same deploy mechanism as #138097).
- Backport targets: `8.19`, `9.3`, `9.4` (branches already carry job_id-only heal from #153399/#153397/#153396).

Complements and extends the runtime heal in #147688.

(cherry picked from commit 6161f77)

# Conflicts:
#	docs/reference/elasticsearch/configuration-reference/machine-learning-settings.md
#	x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/MlIndexTemplateRegistry.java
#	x-pack/qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/MlMappingsUpgradeIT.java

* [ML] Adapt 8.19 backport of #153755 off projects metadata API