Skip to content

feat(tsdb): make ES95 the default TSDB codec - #152861

Merged
salvatore-campagna merged 18 commits into
elastic:mainfrom
salvatore-campagna:feature/es95-default-tsdb-codec
Jul 7, 2026
Merged

salvatore-campagna merged 18 commits into
elastic:mainfrom
salvatore-campagna:feature/es95-default-tsdb-codec

Conversation

@salvatore-campagna

@salvatore-campagna salvatore-campagna commented Jul 3, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR

Make the ES95 TSDB doc values codec the default for time series indices. This removes the es95_codec feature flag and keys the default on a new IndexVersion, TIME_SERIES_ES95_CODEC_DEFAULT, so only indices created after this change use ES95 while existing indices keep their codec. Two settings gate it, a per-index index.time_series.es95_codec.enabled and a cluster wide cluster.time_series.es95_codec.enabled, and both must allow the codec. Closes the feature flag removal item from #141118.

Important

This changes a default. Time series indices created at TIME_SERIES_ES95_CODEC_DEFAULT or later now use the ES95 codec unless index.time_series.es95_codec.enabled or cluster.time_series.es95_codec.enabled is set to false. Reads are unaffected.

Summary

The ES95 pipeline based TSDB doc values codec has been developed behind the es95_codec feature flag. This PR turns it on by default and retires the flag, so new time series indices use it without any explicit setting.

Two settings gate the codec, and both must allow it for a new index to use ES95. The per-index index.time_series.es95_codec.enabled is Final and index scoped; it defaults to true for time series indices created on or after TIME_SERIES_ES95_CODEC_DEFAULT, and setting it to false opts a single index out to ES819. The cluster wide cluster.time_series.es95_codec.enabled is dynamic and acts as a kill switch: when set to false it forces the per-index setting off for new indices on rollover, overruling an explicit per-index opt-in, while existing indices keep the codec they were created with.

The codec is additionally held back on stateless nodes for now. Because stateless.enabled is a node-scoped setting, that check is made in the IndexSettings constructor, where the node settings are available.

Changes

  • IndexSettings: remove the es95_codec flag; default the codec on for time series indices created on or after TIME_SERIES_ES95_CODEC_DEFAULT, and disable it on stateless nodes (checked in the constructor, which has access to node settings).
  • IndexVersions: add TIME_SERIES_ES95_CODEC_DEFAULT.
  • IndexScopedSettings: register the setting unconditionally.
  • ES95CodecClusterSettingProvider: add cluster.time_series.es95_codec.enabled, a dynamic cluster wide opt-out applied at index creation. When set to false it forces the codec off for new time series indices on rollover, overruling an explicit per-index opt-in. Registered through DataStreamsPlugin.
  • TSDBDocValuesFormatSelector: drop the feature flag mention from the javadoc.
  • FeatureFlag (test clusters): drop the ES95_CODEC_FEATURE_FLAG entry.
  • Tests: remove the feature flag assumptions, cover the new default and the stateless holdback, cover the cluster setting gate including its precedence over a per-index opt-in, and opt the ES819 suite out of ES95 explicitly.

Testing

./gradlew :server:test \
  --tests "org.elasticsearch.index.codec.tsdb.TSDBDocValuesFormatSelectorTests" \
  --tests "org.elasticsearch.index.codec.PerFieldMapperCodecTests" \
  --tests "org.elasticsearch.index.codec.tsdb.es95.ES95TSDBDocValuesFormatSingleNodeTests" \
  --tests "org.elasticsearch.index.codec.tsdb.es819.TSDBDocValuesFormatSingleNodeTests" \
  --tests "org.elasticsearch.index.ES95CodecClusterSettingProviderTests"
./gradlew :modules:data-streams:internalClusterTest \
  --tests "org.elasticsearch.datastreams.TSDBES95CodecDefaultIT"
./gradlew :qa:full-cluster-restart:javaRestTest \
  --tests "org.elasticsearch.upgrades.tsdb.TimeSeriesES95FullClusterRestartIT"
./gradlew :qa:rolling-upgrade:javaRestTest \
  --tests "org.elasticsearch.upgrades.TimeSeriesES95RollingUpgradeIT"

Remove the `es95_codec` feature flag and turn the ES95 TSDB doc values
codec on by default for time series indices on stateful nodes. The
`index.time_series.es95_codec.enabled` setting is kept so the codec can
be disabled at index creation.

The default is gated on a new index version,
`TIME_SERIES_ES95_CODEC_DEFAULT_STATEFUL`, so only indices created after
this change use ES95 while existing indices keep ES819.

Serverless is held back: the ES95 codec is disabled on stateless nodes.
Because `stateless.enabled` is a node-scoped setting, this decision is
made in the `IndexSettings` constructor, where the node settings are
available. It will be enabled later through a dedicated
`TIME_SERIES_ES95_CODEC_DEFAULT_STATELESS` index version.
@github-actions

github-actions Bot commented Jul 3, 2026 •

Copy link
Copy Markdown
Contributor

🔍 Preview links for changed docs

⏳ Building and deploying preview... View progress

This comment will be updated with preview links when the build is complete.

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Comment thread docs/changelog/152861.yaml Outdated
type: feature
issues: []
highlight:
title: ES95 is the default TSDB doc values codec on stateful

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
title: ES95 is the default TSDB doc values codec on stateful
title: ES95 is the default TSDB doc values codec

@salvatore-campagna salvatore-campagna Jul 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we remove the detail of stateful as we expect to propagate this to serverless soon after. I was considering fixing that once we have the other PR that removes the serverless gate. But probably if we do this, we avoid another changelog later.

Comment thread docs/changelog/152861.yaml Outdated
highlight:
title: ES95 is the default TSDB doc values codec on stateful
body: |-
New time series indices on stateful Elasticsearch now use the ES95 doc values

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
New time series indices on stateful Elasticsearch now use the ES95 doc values
New time series indices now use the ES95 doc values

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not mention stateful etc in release notes.

public static final IndexVersion TIME_SERIES_ES95_CODEC_DEFAULT_FEATURE_FLAG = def(9_102_0_00, Version.LUCENE_10_4_0);
public static final IndexVersion SEMANTIC_TEXT_ORIGINAL_VALUES_DOC_VALUES = def(9_103_0_00, Version.LUCENE_10_4_0);
public static final IndexVersion UPGRADE_TO_LUCENE_10_5_0 = def(9_104_0_00, Version.LUCENE_10_5_0);
public static final IndexVersion TIME_SERIES_ES95_CODEC_DEFAULT_STATEFUL = def(9_105_0_00, Version.LUCENE_10_5_0);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
public static final IndexVersion TIME_SERIES_ES95_CODEC_DEFAULT_STATEFUL = def(9_105_0_00, Version.LUCENE_10_5_0);
public static final IndexVersion TIME_SERIES_ES95_CODEC_DEFAULT = def(9_105_0_00, Version.LUCENE_10_5_0);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will need another constant once we make the codec default for Serverless too right? This is why I called it TIME_SERIES_ES95_CODEC_DEFAULT_STATEFUL, so that later on we can have TIME_SERIES_ES95_CODEC_DEFAULT_SERVERLESS.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd leave stateful and serverless out of this change, since we can separately control the latter. Actually, let's add a cluster setting to control the use of it, so that we can enable or disable more easily if needed.

@salvatore-campagna salvatore-campagna Jul 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A cluster setting together with the existing index setting? I guess the goal is to make it easier to opt-out for all indices in a cluster...right?

@kkrik-es kkrik-es Jul 3, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, a new index setting and a cluster setting. The latter will make it easy to disable the codec for a project, if need be. But - it can only happen be picked up on a rollover, we can't switch codecs for a given backing index iiuc.

@kkrik-es kkrik-es left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some small changes - no need to mention stateful as it's implied for this repo.

salvatore-campagna and others added 4 commits July 3, 2026 17:19
Add `cluster.time_series.es95_codec.enabled`, a dynamic cluster setting
that disables the ES95 TSDB doc values codec cluster-wide. When set to
`false`, newly created time series indices force
`index.time_series.es95_codec.enabled` to `false`, so the codec is
disabled on rollover while existing indices keep their codec.

The value is applied at index creation through a builtin
`IndexSettingProvider` that caches it from `ClusterSettings`, so a
runtime change is picked up by the next index creation without a
restart.
@salvatore-campagna salvatore-campagna changed the title feat(tsdb): make ES95 the default TSDB codec on stateful feat(tsdb): make ES95 the default TSDB codec Jul 6, 2026
salvatore-campagna and others added 3 commits July 6, 2026 11:37
Rename `TIME_SERIES_ES95_CODEC_DEFAULT_STATEFUL` to
`TIME_SERIES_ES95_CODEC_DEFAULT` and drop `stateful` from the related
test names. The version keys the codec default for all node types; the
stateless holdback is handled separately in the `IndexSettings`
constructor.
Add an integration test that creates a standalone time series index
through the create index API, with `cluster.time_series.es95_codec.enabled`
set to `false`, and asserts the codec is forced off. The existing tests
only exercised the data stream path.
@salvatore-campagna

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-storage-engine (Team:StorageEngine)

@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Hi @salvatore-campagna, I've updated the changelog YAML for you. Note that since this PR is labelled release highlight, you need to update the changelog YAML to fill out the extended information sections.

Comment thread docs/changelog/152861.yaml Outdated
body: |-
New time series indices now use the ES95 doc values codec by default. Set
`index.time_series.es95_codec.enabled` to `false` at index creation to keep the
previous ES819 codec. Existing indices are unaffected.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe outline the improvements of the new codec here, in 1-2 sentences.

salvatore-campagna and others added 5 commits July 6, 2026 12:15
The `release highlight` label bot set `type: non-issue`, which is not in
the changelog schema enum and failed `validateChangelogs`. Use `feature`,
a valid type; combined with the highlight the change appears in both the
features list and the release highlights.
…t-tsdb-codec

# Conflicts:
#	server/src/main/java/org/elasticsearch/index/IndexSettings.java
…t-tsdb-codec

# Conflicts:
#	server/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java
#	server/src/main/java/org/elasticsearch/index/IndexVersions.java
#	test/test-clusters/src/main/java/org/elasticsearch/test/cluster/FeatureFlag.java
@salvatore-campagna
salvatore-campagna merged commit 5c3e509 into elastic:main Jul 7, 2026
37 checks passed
burqen pushed a commit to burqen/elasticsearch that referenced this pull request Jul 7, 2026
* feat(tsdb): make ES95 the default doc values codec

Remove the `es95_codec` feature flag and turn the ES95 TSDB doc values
codec on by default for time series indices on stateful nodes. The
`index.time_series.es95_codec.enabled` setting is kept so the codec can
be disabled at index creation.

The default is gated on a new index version,
`TIME_SERIES_ES95_CODEC_DEFAULT_STATEFUL`, so only indices created after
this change use ES95 while existing indices keep ES819.

Serverless is held back: the ES95 codec is disabled on stateless nodes.
Because `stateless.enabled` is a node-scoped setting, this decision is
made in the `IndexSettings` constructor, where the node settings are
available. It will be enabled later through a dedicated
`TIME_SERIES_ES95_CODEC_DEFAULT_STATELESS` index version.

* docs: add changelog for ES95 default codec on stateful

* docs: drop stateful mention from ES95 changelog

* feat(tsdb): add cluster setting to gate the ES95 codec

Add `cluster.time_series.es95_codec.enabled`, a dynamic cluster setting
that disables the ES95 TSDB doc values codec cluster-wide. When set to
`false`, newly created time series indices force
`index.time_series.es95_codec.enabled` to `false`, so the codec is
disabled on rollover while existing indices keep their codec.

The value is applied at index creation through a builtin
`IndexSettingProvider` that caches it from `ClusterSettings`, so a
runtime change is picked up by the next index creation without a
restart.

* refactor(tsdb): drop stateful from ES95 constant

Rename `TIME_SERIES_ES95_CODEC_DEFAULT_STATEFUL` to
`TIME_SERIES_ES95_CODEC_DEFAULT` and drop `stateful` from the related
test names. The version keys the codec default for all node types; the
stateless holdback is handled separately in the `IndexSettings`
constructor.

* test(tsdb): cover cluster kill switch for standalone index

Add an integration test that creates a standalone time series index
through the create index API, with `cluster.time_series.es95_codec.enabled`
set to `false`, and asserts the codec is forced off. The existing tests
only exercised the data stream path.

* Update docs/changelog/152861.yaml

* docs: set ES95 changelog type to feature

The `release highlight` label bot set `type: non-issue`, which is not in
the changelog schema enum and failed `validateChangelogs`. Use `feature`,
a valid type; combined with the highlight the change appears in both the
features list and the release highlights.

* docs: outline ES95 compression gains in highlight
salvatore-campagna added a commit to salvatore-campagna/elasticsearch that referenced this pull request Jul 7, 2026
elastic#152861 retired the es95_codec feature flag and made ES95 the default
TSDB doc values codec, so the lifecycle IT no longer needs to enable
the flag on its test cluster.
salvatore-campagna added a commit that referenced this pull request Jul 8, 2026
* feat(tsdb): default ES95 codec on serverless

Enable the ES95 TSDB doc values codec by default on stateless nodes,
where #152861 held it back. Drop the blanket stateless holdback in the
`IndexSettings` constructor and gate stateless enablement on a new index
version, `TIME_SERIES_ES95_CODEC_DEFAULT_STATELESS`, so existing
serverless indices keep ES819 and only indices created on or after it
use ES95.

Stateful behavior is unchanged: its default still gates on
`TIME_SERIES_ES95_CODEC_DEFAULT`. The per-index opt-out and the cluster
wide kill switch keep working on serverless, since an explicit `false`
short-circuits the codec regardless of node type.

* refactor(tsdb): drop stateless ES95 version gate

`TIME_SERIES_ES95_CODEC_DEFAULT` has not reached serverless yet, so
gate serverless on it directly and drop the dedicated
`TIME_SERIES_ES95_CODEC_DEFAULT_STATELESS`.
salvatore-campagna added a commit that referenced this pull request Jul 11, 2026
* fix(tsdb): make ES95 codec setting dynamic and classify in CCR

Drop `Property.Final` and add `Property.Dynamic` to
`TIME_SERIES_ES95_CODEC_ENABLED_SETTING` so the setting can be flipped on a
live backing index. This enables the in place migration and rollback paths
exercised by the data stream lifecycle tests, where a single write backing
index switches codec via force merge after a setting update.

Once the setting is dynamic, `TransportResumeFollowActionTests` requires
that it be classified as replicated or non replicated. The codec opt in is
a per index choice and must not follow from leader to follower, so add it
to `NON_REPLICATED_SETTINGS`.

* test(tsdb): add ES95 data stream lifecycle tests

Adds seven REST tests for the ES95 TSDB codec inside TSDB data streams.
Each scenario uses a manual rollover or a dynamic setting flip to exercise
the codec switch path that production users follow when migrating to ES95
in place.

Covered scenarios:
* baseline to ES95 rollover, ES95 to baseline rollover, four way codec
  alternation across rollovers
* in place flip to ES95 on the write backing index followed by force merge
* in place flip back to baseline on the write backing index followed by
  force merge
* alias level queries across mixed codec backing indices
* random access doc values across mixed codec backing indices

Per segment format correctness is covered by `ES95VsES819DocValuesDuelTests`.
Cluster restart and rolling upgrade safety are covered by the matching
javaRestTest suites on their own PRs.

* revert(tsdb): restore Property.Final on ES95 codec setting

Roll back `TIME_SERIES_ES95_CODEC_ENABLED_SETTING` to its original
`Property.IndexScope` plus `Property.Final` after review feedback
preferring the historical convention of immutable codec settings.
The setting is fixed at index creation and cannot be updated on a
live index.

Drop the CCR `NON_REPLICATED_SETTINGS` entry, only required while the
setting was dynamic, and the two tests that flip the setting on a
live write backing index: `testInPlaceFlipToES95OnWriteBackingIndex`
and `testInPlaceFlipToBaselineOnWriteBackingIndex`. The
`updateIndexSetting` and `forceMerge` helpers have no remaining
callers and are removed with them.

* test(tsdb): wire ES95 codec feature flag in data stream lifecycle cluster

* test(tsdb): drop es95_codec flag from lifecycle IT

#152861 retired the es95_codec feature flag and made ES95 the default
TSDB doc values codec, so the lifecycle IT no longer needs to enable
the flag on its test cluster.

* test(tsdb): make ES95 lifecycle IT stateless-safe

The serverless CI re-runs this IT against a stateless cluster, where a
new backing index's search shards are loaded from the object store
asynchronously. The count read at the start of bulkIndex raced that
loading and failed with a 503 no_shard_available_action_exception.

Wait for the write index with ensureGreen before that read.

* test(tsdb): drop replica pin in ES95 lifecycle IT

The serverless CI re-runs this IT against a stateless cluster, where
reads are served by a separate search shard. Pinning
index.number_of_replicas to 0 left the search node without a shard, so
the count and search calls failed with a 503
no_shard_available_action_exception.

Drop the pin so stateless can provision the search shard.

* test(tsdb): wait for a searchable shard, not green, in ES95 IT

Dropping the replica pin fixed serverless but broke the single node
stateful run: the default replica stays unassigned, the cluster sits at
yellow, and ensureGreen times out on every backing index.

Replace ensureGreen with a probe that retries a search until the shard
can serve it. This fits both cluster shapes: stateful serves from the
primary at once, and stateless retries until its asynchronously
provisioned search shard is ready. Cap the wait at 60s so a stuck shard
fails the test instead of blocking.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants