Repository navigation
feat(tsdb): make ES95 the default TSDB codec - #152861
salvatore-campagna merged 18 commits into
Conversation
Remove the `es95_codec` feature flag and turn the ES95 TSDB doc values codec on by default for time series indices on stateful nodes. The `index.time_series.es95_codec.enabled` setting is kept so the codec can be disabled at index creation. The default is gated on a new index version, `TIME_SERIES_ES95_CODEC_DEFAULT_STATEFUL`, so only indices created after this change use ES95 while existing indices keep ES819. Serverless is held back: the ES95 codec is disabled on stateless nodes. Because `stateless.enabled` is a node-scoped setting, this decision is made in the `IndexSettings` constructor, where the node settings are available. It will be enabled later through a dedicated `TIME_SERIES_ES95_CODEC_DEFAULT_STATELESS` index version.
🔍 Preview links for changed docs⏳ Building and deploying preview... View progress This comment will be updated with preview links when the build is complete. |
ℹ️ Important: Docs version tagging👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version. We use applies_to tags to mark version-specific features and changes. Expand for a quick overviewWhen to use applies_to tags:✅ At the page level to indicate which products/deployments the content applies to (mandatory) What NOT to do:❌ Don't remove or replace information that applies to an older version 🤔 Need help?
|
| type: feature | ||
| issues: [] | ||
| highlight: | ||
| title: ES95 is the default TSDB doc values codec on stateful |
There was a problem hiding this comment.
| title: ES95 is the default TSDB doc values codec on stateful | |
| title: ES95 is the default TSDB doc values codec |
There was a problem hiding this comment.
I guess we remove the detail of stateful as we expect to propagate this to serverless soon after. I was considering fixing that once we have the other PR that removes the serverless gate. But probably if we do this, we avoid another changelog later.
| highlight: | ||
| title: ES95 is the default TSDB doc values codec on stateful | ||
| body: |- | ||
| New time series indices on stateful Elasticsearch now use the ES95 doc values |
There was a problem hiding this comment.
| New time series indices on stateful Elasticsearch now use the ES95 doc values | |
| New time series indices now use the ES95 doc values |
There was a problem hiding this comment.
Let's not mention stateful etc in release notes.
| public static final IndexVersion TIME_SERIES_ES95_CODEC_DEFAULT_FEATURE_FLAG = def(9_102_0_00, Version.LUCENE_10_4_0); | ||
| public static final IndexVersion SEMANTIC_TEXT_ORIGINAL_VALUES_DOC_VALUES = def(9_103_0_00, Version.LUCENE_10_4_0); | ||
| public static final IndexVersion UPGRADE_TO_LUCENE_10_5_0 = def(9_104_0_00, Version.LUCENE_10_5_0); | ||
| public static final IndexVersion TIME_SERIES_ES95_CODEC_DEFAULT_STATEFUL = def(9_105_0_00, Version.LUCENE_10_5_0); |
There was a problem hiding this comment.
| public static final IndexVersion TIME_SERIES_ES95_CODEC_DEFAULT_STATEFUL = def(9_105_0_00, Version.LUCENE_10_5_0); | |
| public static final IndexVersion TIME_SERIES_ES95_CODEC_DEFAULT = def(9_105_0_00, Version.LUCENE_10_5_0); |
There was a problem hiding this comment.
We will need another constant once we make the codec default for Serverless too right? This is why I called it TIME_SERIES_ES95_CODEC_DEFAULT_STATEFUL, so that later on we can have TIME_SERIES_ES95_CODEC_DEFAULT_SERVERLESS.
There was a problem hiding this comment.
I'd leave stateful and serverless out of this change, since we can separately control the latter. Actually, let's add a cluster setting to control the use of it, so that we can enable or disable more easily if needed.
There was a problem hiding this comment.
A cluster setting together with the existing index setting? I guess the goal is to make it easier to opt-out for all indices in a cluster...right?
There was a problem hiding this comment.
Correct, a new index setting and a cluster setting. The latter will make it easy to disable the codec for a project, if need be. But - it can only happen be picked up on a rollover, we can't switch codecs for a given backing index iiuc.
kkrik-es
left a comment
There was a problem hiding this comment.
Some small changes - no need to mention stateful as it's implied for this repo.
Add `cluster.time_series.es95_codec.enabled`, a dynamic cluster setting that disables the ES95 TSDB doc values codec cluster-wide. When set to `false`, newly created time series indices force `index.time_series.es95_codec.enabled` to `false`, so the codec is disabled on rollover while existing indices keep their codec. The value is applied at index creation through a builtin `IndexSettingProvider` that caches it from `ClusterSettings`, so a runtime change is picked up by the next index creation without a restart.
ES95 the default TSDB codec on statefulES95 the default TSDB codec
Rename `TIME_SERIES_ES95_CODEC_DEFAULT_STATEFUL` to `TIME_SERIES_ES95_CODEC_DEFAULT` and drop `stateful` from the related test names. The version keys the codec default for all node types; the stateless holdback is handled separately in the `IndexSettings` constructor.
Add an integration test that creates a standalone time series index through the create index API, with `cluster.time_series.es95_codec.enabled` set to `false`, and asserts the codec is forced off. The existing tests only exercised the data stream path.
|
@coderabbitai review |
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
|
Hi @salvatore-campagna, I've updated the changelog YAML for you. Note that since this PR is labelled |
| body: |- | ||
| New time series indices now use the ES95 doc values codec by default. Set | ||
| `index.time_series.es95_codec.enabled` to `false` at index creation to keep the | ||
| previous ES819 codec. Existing indices are unaffected. |
There was a problem hiding this comment.
Maybe outline the improvements of the new codec here, in 1-2 sentences.
The `release highlight` label bot set `type: non-issue`, which is not in the changelog schema enum and failed `validateChangelogs`. Use `feature`, a valid type; combined with the highlight the change appears in both the features list and the release highlights.
…t-tsdb-codec # Conflicts: # server/src/main/java/org/elasticsearch/index/IndexSettings.java
…t-tsdb-codec # Conflicts: # server/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java # server/src/main/java/org/elasticsearch/index/IndexVersions.java # test/test-clusters/src/main/java/org/elasticsearch/test/cluster/FeatureFlag.java
* feat(tsdb): make ES95 the default doc values codec Remove the `es95_codec` feature flag and turn the ES95 TSDB doc values codec on by default for time series indices on stateful nodes. The `index.time_series.es95_codec.enabled` setting is kept so the codec can be disabled at index creation. The default is gated on a new index version, `TIME_SERIES_ES95_CODEC_DEFAULT_STATEFUL`, so only indices created after this change use ES95 while existing indices keep ES819. Serverless is held back: the ES95 codec is disabled on stateless nodes. Because `stateless.enabled` is a node-scoped setting, this decision is made in the `IndexSettings` constructor, where the node settings are available. It will be enabled later through a dedicated `TIME_SERIES_ES95_CODEC_DEFAULT_STATELESS` index version. * docs: add changelog for ES95 default codec on stateful * docs: drop stateful mention from ES95 changelog * feat(tsdb): add cluster setting to gate the ES95 codec Add `cluster.time_series.es95_codec.enabled`, a dynamic cluster setting that disables the ES95 TSDB doc values codec cluster-wide. When set to `false`, newly created time series indices force `index.time_series.es95_codec.enabled` to `false`, so the codec is disabled on rollover while existing indices keep their codec. The value is applied at index creation through a builtin `IndexSettingProvider` that caches it from `ClusterSettings`, so a runtime change is picked up by the next index creation without a restart. * refactor(tsdb): drop stateful from ES95 constant Rename `TIME_SERIES_ES95_CODEC_DEFAULT_STATEFUL` to `TIME_SERIES_ES95_CODEC_DEFAULT` and drop `stateful` from the related test names. The version keys the codec default for all node types; the stateless holdback is handled separately in the `IndexSettings` constructor. * test(tsdb): cover cluster kill switch for standalone index Add an integration test that creates a standalone time series index through the create index API, with `cluster.time_series.es95_codec.enabled` set to `false`, and asserts the codec is forced off. The existing tests only exercised the data stream path. * Update docs/changelog/152861.yaml * docs: set ES95 changelog type to feature The `release highlight` label bot set `type: non-issue`, which is not in the changelog schema enum and failed `validateChangelogs`. Use `feature`, a valid type; combined with the highlight the change appears in both the features list and the release highlights. * docs: outline ES95 compression gains in highlight
elastic#152861 retired the es95_codec feature flag and made ES95 the default TSDB doc values codec, so the lifecycle IT no longer needs to enable the flag on its test cluster.
* feat(tsdb): default ES95 codec on serverless Enable the ES95 TSDB doc values codec by default on stateless nodes, where #152861 held it back. Drop the blanket stateless holdback in the `IndexSettings` constructor and gate stateless enablement on a new index version, `TIME_SERIES_ES95_CODEC_DEFAULT_STATELESS`, so existing serverless indices keep ES819 and only indices created on or after it use ES95. Stateful behavior is unchanged: its default still gates on `TIME_SERIES_ES95_CODEC_DEFAULT`. The per-index opt-out and the cluster wide kill switch keep working on serverless, since an explicit `false` short-circuits the codec regardless of node type. * refactor(tsdb): drop stateless ES95 version gate `TIME_SERIES_ES95_CODEC_DEFAULT` has not reached serverless yet, so gate serverless on it directly and drop the dedicated `TIME_SERIES_ES95_CODEC_DEFAULT_STATELESS`.
* fix(tsdb): make ES95 codec setting dynamic and classify in CCR Drop `Property.Final` and add `Property.Dynamic` to `TIME_SERIES_ES95_CODEC_ENABLED_SETTING` so the setting can be flipped on a live backing index. This enables the in place migration and rollback paths exercised by the data stream lifecycle tests, where a single write backing index switches codec via force merge after a setting update. Once the setting is dynamic, `TransportResumeFollowActionTests` requires that it be classified as replicated or non replicated. The codec opt in is a per index choice and must not follow from leader to follower, so add it to `NON_REPLICATED_SETTINGS`. * test(tsdb): add ES95 data stream lifecycle tests Adds seven REST tests for the ES95 TSDB codec inside TSDB data streams. Each scenario uses a manual rollover or a dynamic setting flip to exercise the codec switch path that production users follow when migrating to ES95 in place. Covered scenarios: * baseline to ES95 rollover, ES95 to baseline rollover, four way codec alternation across rollovers * in place flip to ES95 on the write backing index followed by force merge * in place flip back to baseline on the write backing index followed by force merge * alias level queries across mixed codec backing indices * random access doc values across mixed codec backing indices Per segment format correctness is covered by `ES95VsES819DocValuesDuelTests`. Cluster restart and rolling upgrade safety are covered by the matching javaRestTest suites on their own PRs. * revert(tsdb): restore Property.Final on ES95 codec setting Roll back `TIME_SERIES_ES95_CODEC_ENABLED_SETTING` to its original `Property.IndexScope` plus `Property.Final` after review feedback preferring the historical convention of immutable codec settings. The setting is fixed at index creation and cannot be updated on a live index. Drop the CCR `NON_REPLICATED_SETTINGS` entry, only required while the setting was dynamic, and the two tests that flip the setting on a live write backing index: `testInPlaceFlipToES95OnWriteBackingIndex` and `testInPlaceFlipToBaselineOnWriteBackingIndex`. The `updateIndexSetting` and `forceMerge` helpers have no remaining callers and are removed with them. * test(tsdb): wire ES95 codec feature flag in data stream lifecycle cluster * test(tsdb): drop es95_codec flag from lifecycle IT #152861 retired the es95_codec feature flag and made ES95 the default TSDB doc values codec, so the lifecycle IT no longer needs to enable the flag on its test cluster. * test(tsdb): make ES95 lifecycle IT stateless-safe The serverless CI re-runs this IT against a stateless cluster, where a new backing index's search shards are loaded from the object store asynchronously. The count read at the start of bulkIndex raced that loading and failed with a 503 no_shard_available_action_exception. Wait for the write index with ensureGreen before that read. * test(tsdb): drop replica pin in ES95 lifecycle IT The serverless CI re-runs this IT against a stateless cluster, where reads are served by a separate search shard. Pinning index.number_of_replicas to 0 left the search node without a shard, so the count and search calls failed with a 503 no_shard_available_action_exception. Drop the pin so stateless can provision the search shard. * test(tsdb): wait for a searchable shard, not green, in ES95 IT Dropping the replica pin fixed serverless but broke the single node stateful run: the default replica stays unassigned, the cluster sits at yellow, and ensureGreen times out on every backing index. Replace ensureGreen with a probe that retries a search until the shard can serve it. This fits both cluster shapes: stateful serves from the primary at once, and stateless retries until its asynchronously provisioned search shard is ready. Cap the wait at 60s so a stuck shard fails the test instead of blocking.
TL;DR
Make the
ES95TSDB doc values codec the default for time series indices. This removes thees95_codecfeature flag and keys the default on a newIndexVersion,TIME_SERIES_ES95_CODEC_DEFAULT, so only indices created after this change useES95while existing indices keep their codec. Two settings gate it, a per-indexindex.time_series.es95_codec.enabledand a cluster widecluster.time_series.es95_codec.enabled, and both must allow the codec. Closes the feature flag removal item from #141118.Important
This changes a default. Time series indices created at
TIME_SERIES_ES95_CODEC_DEFAULTor later now use theES95codec unlessindex.time_series.es95_codec.enabledorcluster.time_series.es95_codec.enabledis set tofalse. Reads are unaffected.Summary
The
ES95pipeline based TSDB doc values codec has been developed behind thees95_codecfeature flag. This PR turns it on by default and retires the flag, so new time series indices use it without any explicit setting.Two settings gate the codec, and both must allow it for a new index to use
ES95. The per-indexindex.time_series.es95_codec.enabledisFinaland index scoped; it defaults totruefor time series indices created on or afterTIME_SERIES_ES95_CODEC_DEFAULT, and setting it tofalseopts a single index out toES819. The cluster widecluster.time_series.es95_codec.enabledis dynamic and acts as a kill switch: when set tofalseit forces the per-index setting off for new indices on rollover, overruling an explicit per-index opt-in, while existing indices keep the codec they were created with.The codec is additionally held back on stateless nodes for now. Because
stateless.enabledis a node-scoped setting, that check is made in theIndexSettingsconstructor, where the node settings are available.Changes
IndexSettings: remove thees95_codecflag; default the codec on for time series indices created on or afterTIME_SERIES_ES95_CODEC_DEFAULT, and disable it on stateless nodes (checked in the constructor, which has access to node settings).IndexVersions: addTIME_SERIES_ES95_CODEC_DEFAULT.IndexScopedSettings: register the setting unconditionally.ES95CodecClusterSettingProvider: addcluster.time_series.es95_codec.enabled, a dynamic cluster wide opt-out applied at index creation. When set tofalseit forces the codec off for new time series indices on rollover, overruling an explicit per-index opt-in. Registered throughDataStreamsPlugin.TSDBDocValuesFormatSelector: drop the feature flag mention from the javadoc.FeatureFlag(test clusters): drop theES95_CODEC_FEATURE_FLAGentry.ES819suite out ofES95explicitly.Testing