Fix prefix escape for synthetic id by dnhatn · Pull Request #150433 · elastic/elasticsearch

dnhatn · 2026-06-02T01:48:39Z

We have an ES|QL query running for many hours on a tsdb index that is just 32M docs / 83MB - this should complete within seconds.

FROM metrics-k8sclusterreceiver.otel-default METADATA _id, _source
| WHERE k8s.deployment.available < k8s.deployment.desired
        AND _id NOT IN ()
| LIMIT 1000

_id: [fd ff c4 6f 4b 15 68 a0 8a 1d 7 b1 38 5f 3f 29 9a 7f ff fe 61 9c de bc b8 87 7a b5 b1]

Some ids start with 0xfd, a special escape byte that is prepended by Uid#encodeId when the first byte is greater than 0xfd, but SyntheticIdTermsEnum#seekCeil doesn't strip it.

Here is how the infinite loop occurs:

AbstractMultiTermQueryConstantScoreWrapper#collectTerms with [fd ff c4 ...]
FilteredTermsEnum#next calls seekCeil with [fd ff c4 ...]
SyntheticIdTermsEnum#seekCeil extracts the tsid but includes the 0xfd escape prefix, producing [fd ff c4 ...] instead of the actual tsid [ff c4 ...]
SyntheticIdTermsEnum#seekCeil positions at a term greater than [fd ff c4 ...], but docSyntheticId synthesizes the Lucene term by prepending 0xfd - making it smaller than the seeking term. This violates the seekCeil contract.
FilteredTermsEnum#next sees the positioned term is still before the seeking term and calls seekCeil again - repeating steps 2–4 infinitely.

"elasticsearch[es-es-search-79c95b95f4-t6qlg][esql_worker][T#7]" [42 frames]
org.elasticsearch.index.codec.tsdb.TSDBSyntheticIdDocValuesHolder.lookupTsIdTerm(TSDBSyntheticIdDocValuesHolder.java:155)
  at org.elasticsearch.index.codec.tsdb.TSDBSyntheticIdFieldsProducer$SyntheticIdTermsEnum.seekCeil(TSDBSyntheticIdFieldsProducer.java:211)
  at org.elasticsearch.index.codec.bloomfilter.LazyFilterTermsEnum.seekCeil(LazyFilterTermsEnum.java:27)
  at org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.seekCeil(FilterLeafReader.java:189)
  at org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.seekCeil(FilterLeafReader.java:189)
  at org.apache.lucene.index.FilteredTermsEnum.next(FilteredTermsEnum.java:236)
  at org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.collectTerms(AbstractMultiTermQueryConstantScoreWrapper.java:175)
  at org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.scorerSupplier(AbstractMultiTermQueryConstantScoreWrapper.java:229)
  at org.elasticsearch.lucene.search.XLRUQueryCache$CachingWrapperWeight.scorerSupplier(XLRUQueryCache.java:749)
  at org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:243)
  at org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:290)
  at org.elasticsearch.lucene.search.XLRUQueryCache$CachingWrapperWeight.scorerSupplier(XLRUQueryCache.java:749)
  at org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:243)
  at org.apache.lucene.search.Weight.bulkScorer(Weight.java:171)
  at org.elasticsearch.compute.lucene.query.LuceneOperator$LuceneScorer.reinitialize(LuceneOperator.java:346)
  at org.elasticsearch.compute.lucene.query.LuceneOperator.getCurrentOrLoadNextScorer(LuceneOperator.java:223)
  at org.elasticsearch.compute.lucene.query.LuceneSourceOperator.getCheckedOutput(LuceneSourceOperator.java:299)
  at org.elasticsearch.compute.lucene.query.LuceneOperator.getOutput(LuceneOperator.java:175)
  at org.elasticsearch.compute.operator.Driver.runSingleLoopIteration(Driver.java:323)

Closes #150389
Relates #145018

elasticsearchmachine · 2026-06-02T01:56:26Z

Hi @dnhatn, I've created a changelog YAML for you.

elasticsearchmachine · 2026-06-02T01:56:49Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

github-actions · 2026-06-02T01:57:13Z

🔍 Preview links for changed docs

⏳ Building and deploying preview... View progress

This comment will be updated with preview links when the build is complete.

github-actions · 2026-06-02T01:59:24Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

martijnvg

Good catch Nhat, LGTM

fcofdez

LGTM. Great catch Nhat!

tlrx

LGTM, thanks for fixing this Nhat!

For your information and in case you haven't seen it, I fixed an escape char related issue in #145018. I think both are complementary and your PR here fixes the handling of the escape char on the input (seekCeil) path, while mine fixed on the output (term) path. Hopefully it's correct everywhere now :)

tlrx · 2026-06-02T07:18:42Z

-                // Lookup whatever term `id` has been provided
-                tsIdOrd = docValues.lookupTsIdTerm(id);
+        private static BytesRef extractTsid(BytesRef id) {
+            final int escapeBytes = id.length > 0 && Byte.toUnsignedInt(id.bytes[id.offset]) >= Uid.BASE64_ESCAPE ? 1 : 0;


Can we add a bit of documentation explaining what the escape what is and why it needs special handling here?

added a comment in 34eea32

tlrx · 2026-06-02T07:32:55Z

    }

+    // See #createSyntheticId
    public static BytesRef extractTimeSeriesIdFromSyntheticId(BytesRef id) {


This one is also used in ParsedDocument. I think it has the nice effect to also strip the escape char for the _tsid that is set to tombstones documents, and that fixes a double-escaping of the value indexed in Lucene.

It also fixes some tests too :)

tlrx · 2026-06-02T07:33:24Z

    }

+    public void testSyntheticId() {
+        for (int prefix = 0; prefix <= 0xff; prefix++) {


Good idea to just test all values 👍

dnhatn · 2026-06-02T18:28:44Z

Thanks friends!

elasticsearchmachine · 2026-06-02T18:30:23Z

💚 Backport successful

Status	Branch	Result
✅	9.4

We have an ES|QL query running for many hours on a tsdb index that is just 32M docs / 83MB - this should complete within seconds. ```sql FROM metrics-k8sclusterreceiver.otel-default METADATA _id, _source | WHERE k8s.deployment.available < k8s.deployment.desired AND _id NOT IN () | LIMIT 1000 ``` ``` _id: [fd ff c4 6f 4b 15 68 a0 8a 1d 7 b1 38 5f 3f 29 9a 7f ff fe 61 9c de bc b8 87 7a b5 b1] ``` Some ids start with `0xfd`, a special escape byte that is prepended by `Uid#encodeId` when the first byte is greater than `0xfd`, but `SyntheticIdTermsEnum#seekCeil` doesn't strip it. Here is how the infinite loop occurs: 1. `AbstractMultiTermQueryConstantScoreWrapper#collectTerms` with `[fd ff c4 ...]` 2. `FilteredTermsEnum#next` calls `seekCeil` with `[fd ff c4 ...]` 3. `SyntheticIdTermsEnum#seekCeil` extracts the tsid but includes the `0xfd` escape prefix, producing `[fd ff c4 ...]` instead of the actual tsid `[ff c4 ...]` 4. `SyntheticIdTermsEnum#seekCeil` positions at a term greater than `[fd ff c4 ...]`, but `docSyntheticId` synthesizes the Lucene term by prepending `0xfd` - making it smaller than the seeking term. This violates the seekCeil contract. 5. `FilteredTermsEnum#next` sees the positioned term is still before the seeking term and calls seekCeil again - repeating steps 2–4 infinitely.

tylerperk · 2026-06-03T12:27:16Z

Thanks @dnhatn - please backport this as far as you can, where applicable. 9.3 at minimum.

kkrik-es · 2026-06-03T12:54:12Z

Thanks @dnhatn - please backport this as far as you can, where applicable. 9.3 at minimum.

synthetic id was introduced in 9.4, I think.

dnhatn · 2026-06-03T14:15:07Z

Yes, I was introduced in 9.4.0.

We have an ES|QL query running for many hours on a tsdb index that is just 32M docs / 83MB - this should complete within seconds. ```sql FROM metrics-k8sclusterreceiver.otel-default METADATA _id, _source | WHERE k8s.deployment.available < k8s.deployment.desired AND _id NOT IN () | LIMIT 1000 ``` ``` _id: [fd ff c4 6f 4b 15 68 a0 8a 1d 7 b1 38 5f 3f 29 9a 7f ff fe 61 9c de bc b8 87 7a b5 b1] ``` Some ids start with `0xfd`, a special escape byte that is prepended by `Uid#encodeId` when the first byte is greater than `0xfd`, but `SyntheticIdTermsEnum#seekCeil` doesn't strip it. Here is how the infinite loop occurs: 1. `AbstractMultiTermQueryConstantScoreWrapper#collectTerms` with `[fd ff c4 ...]` 2. `FilteredTermsEnum#next` calls `seekCeil` with `[fd ff c4 ...]` 3. `SyntheticIdTermsEnum#seekCeil` extracts the tsid but includes the `0xfd` escape prefix, producing `[fd ff c4 ...]` instead of the actual tsid `[ff c4 ...]` 4. `SyntheticIdTermsEnum#seekCeil` positions at a term greater than `[fd ff c4 ...]`, but `docSyntheticId` synthesizes the Lucene term by prepending `0xfd` - making it smaller than the seeking term. This violates the seekCeil contract. 5. `FilteredTermsEnum#next` sees the positioned term is still before the seeking term and calls seekCeil again - repeating steps 2–4 infinitely.

The prefix escape fix for synthetic id in #150433 is not enough for two cases: 1. The prefix starts with 0xFE or 0xFF - these terms never exist and we can safely return END. Currently, seekCeil(0xFF_00) calls seekCeil(0x00) internally and the returned term is before 0xFF_00, breaking the seekCeil contract and leading to an infinite loop. 2. The prefix starts with 0xFD, but the second byte is less than 0xFD. For example: 0xFD_00 - currently, we call seekCeil(0x00) internally and the found term (e.g. 0x00_01) is less than 0xFD_00 because no escape prefix is prepended, leading the returned term to be smaller than the seeking term, breaking the seekCeil contract. #150433 focused on valid terms; this change covers invalid terms. Relates #145018 Relates #150433

Fix prefix escape for synthetic id

ff3d0af

elasticsearchmachine added the v9.5.0 label Jun 2, 2026

dnhatn added v9.4.3 :StorageEngine/TSDB You know, for Metrics >bug auto-backport Automatically create backport pull requests when merged labels Jun 2, 2026

dnhatn requested review from fcofdez and tlrx June 2, 2026 01:55

[CI] Auto commit changes from spotless

6985552

dnhatn requested a review from martijnvg June 2, 2026 01:55

dnhatn marked this pull request as ready for review June 2, 2026 01:56

Update docs/changelog/150433.yaml

7832b11

elasticsearchmachine added the Team:StorageEngine label Jun 2, 2026

martijnvg approved these changes Jun 2, 2026

View reviewed changes

fcofdez approved these changes Jun 2, 2026

View reviewed changes

tlrx approved these changes Jun 2, 2026

View reviewed changes

dnhatn added 2 commits June 2, 2026 10:02

Merge remote-tracking branch 'elastic/main' into synthetic-id

decb44d

comment

34eea32

dnhatn merged commit 3173e6f into elastic:main Jun 2, 2026
36 checks passed

dnhatn deleted the synthetic-id branch June 2, 2026 18:29

dnhatn mentioned this pull request Jun 2, 2026

[9.4] Fix prefix escape for synthetic id (#150433) #150543

Merged

dnhatn mentioned this pull request Jun 19, 2026

Harden synthetic id escape prefix #151696

Merged

dnhatn mentioned this pull request Jun 19, 2026

Harden synthetic id escape prefix #151695

Merged

Conversation

dnhatn commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 2, 2026

Uh oh!

elasticsearchmachine commented Jun 2, 2026

Uh oh!

github-actions Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

github-actions Bot commented Jun 2, 2026

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

martijnvg left a comment

Choose a reason for hiding this comment

Uh oh!

fcofdez left a comment

Choose a reason for hiding this comment

Uh oh!

tlrx left a comment

Choose a reason for hiding this comment

Uh oh!

tlrx Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

dnhatn Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

tlrx Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

tlrx Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

dnhatn commented Jun 2, 2026

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 2, 2026

💚 Backport successful

Uh oh!

tylerperk commented Jun 3, 2026

Uh oh!

kkrik-es commented Jun 3, 2026

Uh oh!

dnhatn commented Jun 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

dnhatn commented Jun 2, 2026 •

edited

Loading

github-actions Bot commented Jun 2, 2026 •

edited

Loading