Skip to content

Apply source filter to metadata field mappers when loading synthetic source - #143726

Merged
gmarouli merged 13 commits into
elastic:mainfrom
gmarouli:only-keep-ignored-source-with-source-filter
Mar 9, 2026
Merged

gmarouli merged 13 commits into
elastic:mainfrom
gmarouli:only-keep-ignored-source-with-source-filter

Conversation

@gmarouli

@gmarouli gmarouli commented Mar 6, 2026 •

Copy link
Copy Markdown
Contributor

When loading synthetic fields we use the SourceFilter to determine which ones will be included. However, ObjectMapper includes all the metadata fields as well.

Usually, this is not an issue because most metadata fields do not support synthetic source.

However, the _doc_count does and when we were displaying the dimensions in ES|QL of a downsampled index where all docs have the _doc_count set, it adds _doc_count in the output of _timeseries.

In this PR, we change the innerSyntheticFieldLoader of the ObjectMapper to apply the filter also on the metadata. We do always include the IgnoredSourceFieldMapper because we need it to load the fields from it.

Closes #143464

@coderabbitai

coderabbitai Bot commented Mar 6, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (2)
  • Team:Delivery
  • Team:Search - Inference

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: 813f9cde-0710-471d-bea9-56af589fc4a3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Comment @coderabbitai help to get the list of available commands and usage tips.

@gmarouli gmarouli changed the title Filter metadata field mappers when source filter is set [WIP] Filter metadata field mappers when source filter is set Mar 6, 2026
@gmarouli gmarouli added >bug :StorageEngine/Mapping The storage related side of mappings branch:9.3 labels Mar 6, 2026
@gmarouli
gmarouli requested a review from romseygeek March 6, 2026 11:34
@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Hi @gmarouli, I've created a changelog YAML for you.

@gmarouli gmarouli changed the title [WIP] Filter metadata field mappers when source filter is set Apply source filter to metadata field mappers when loading synthetic source Mar 6, 2026
@gmarouli
gmarouli marked this pull request as ready for review March 6, 2026 11:38
@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-storage-engine (Team:StorageEngine)

@romseygeek romseygeek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Comment thread server/src/main/java/org/elasticsearch/search/lookup/SourceFilter.java Outdated
@gmarouli gmarouli added auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) auto-backport Automatically create backport pull requests when merged labels Mar 9, 2026
@gmarouli
gmarouli merged commit 72c2244 into elastic:main Mar 9, 2026
33 of 36 checks passed
@gmarouli
gmarouli deleted the only-keep-ignored-source-with-source-filter branch March 9, 2026 12:08
@gmarouli

gmarouli commented Mar 9, 2026

Copy link
Copy Markdown
Contributor Author

💚 All backports created successfully

Status Branch Result
✅ 9.3

Questions ?

Please refer to the Backport tool documentation

gmarouli added a commit to gmarouli/elasticsearch that referenced this pull request Mar 9, 2026
…source (elastic#143726)

When loading synthetic fields we use the `SourceFilter` to determine which ones will be included. However, `ObjectMapper` includes all the metadata fields as well.

Usually, this is not an issue because most metadata fields do not support synthetic source.

However, the `_doc_count` does and when we were displaying the dimensions in ES|QL of a downsampled index where all docs have the `_doc_count` set, it adds `_doc_count` in the output of `_timeseries`.

In this PR, we change the `innerSyntheticFieldLoader` of the `ObjectMapper` to apply the filter also on the metadata. We do always include the `IgnoredSourceFieldMapper` because we need it to load the fields from it.

Closes elastic#143464

(cherry picked from commit 72c2244)

# Conflicts:
#	x-pack/plugin/esql/qa/testFixtures/src/main/resources/k8s-timeseries.csv-spec
#	x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java
gmarouli added a commit that referenced this pull request Mar 9, 2026
…source (#143726) (#143851)

When loading synthetic fields we use the `SourceFilter` to determine which ones will be included. However, `ObjectMapper` includes all the metadata fields as well.

Usually, this is not an issue because most metadata fields do not support synthetic source.

However, the `_doc_count` does and when we were displaying the dimensions in ES|QL of a downsampled index where all docs have the `_doc_count` set, it adds `_doc_count` in the output of `_timeseries`.

In this PR, we change the `innerSyntheticFieldLoader` of the `ObjectMapper` to apply the filter also on the metadata. We do always include the `IgnoredSourceFieldMapper` because we need it to load the fields from it.

Closes #143464

(cherry picked from commit 72c2244)

# Conflicts:
#	x-pack/plugin/esql/qa/testFixtures/src/main/resources/k8s-timeseries.csv-spec
#	x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-backport Automatically create backport pull requests when merged auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) >bug :StorageEngine/Mapping The storage related side of mappings Team:StorageEngine v9.3.2 v9.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

_timeseries metadata field on downsampled indices includes _doc_count

3 participants