Attachment node and processor cap settings by kingherc · Pull Request #148493 · elastic/elasticsearch

kingherc · 2026-05-07T11:29:22Z

Processor setting to cap decoded attachment input before Tika. Optional node-level cap setting as well. Both default to unset, for backwards compatibility.

Also introduces metrics for attachment raw sizes before the cap check and for attachments that complete processing.

TODOs after this PR is merged:

extend Processors.ts in elastic/elasticsearch-specification if
it is not done automatically.
see about AttachmentProcessor.java being regenerated in elastic/elasticsearch-java

Relates #97819

elasticsearchmachine · 2026-05-07T11:29:54Z

Hi @kingherc, I've created a changelog YAML for you.

github-actions · 2026-05-07T11:32:38Z

🔍 Preview links for changed docs

docs/reference/enrich-processor/attachment.md

github-actions · 2026-05-07T11:32:41Z

✅ Vale Linting Results

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

github-actions · 2026-05-07T11:34:21Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

Copilot

Pull request overview

Adds configurable limits to the ingest attachment processor to cap decoded attachment payload size before handing bytes to Tika, via both a per-processor option and an optional node-level default, with accompanying tests and changelog/docs updates.

Changes:

Add max_attachment_bytes processor option and enforce decoded-input size limits in AttachmentProcessor.
Introduce ingest.attachment.max_attachment_size node setting (relative/absolute) and wire settings through the plugin/factory.
Add REST/YAML and unit tests for rejection behavior and on-failure handling; update module test cluster dependencies.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
modules/ingest-attachment/src/yamlRestTest/resources/rest-api-spec/test/ingest_attachment/20_max_input_bytes.yml	New REST tests covering max decoded input size behavior and on-failure.
modules/ingest-attachment/src/yamlRestTest/java/org/elasticsearch/ingest/attachment/IngestAttachmentClientYamlTestSuiteIT.java	Test cluster now includes `ingest-common` for `set` processor usage in on_failure.
modules/ingest-attachment/src/test/java/org/elasticsearch/ingest/attachment/AttachmentProcessorTests.java	Unit tests for per-processor and node-level caps (absolute/ratio).
modules/ingest-attachment/src/test/java/org/elasticsearch/ingest/attachment/AttachmentProcessorFactoryTests.java	Factory tests updated for node settings and invalid `max_attachment_bytes`.
modules/ingest-attachment/src/main/resources/META-INF/services/org.elasticsearch.features.FeatureSpecification	Registers ingest-attachment feature specification for tests.
modules/ingest-attachment/src/main/java/org/elasticsearch/ingest/attachment/IngestAttachmentPluginFeatures.java	Defines the attachment max-size cluster feature for test gating.
modules/ingest-attachment/src/main/java/org/elasticsearch/ingest/attachment/IngestAttachmentPlugin.java	Registers the new node setting and passes node settings to the processor factory.
modules/ingest-attachment/src/main/java/org/elasticsearch/ingest/attachment/AttachmentProcessor.java	Implements node setting, processor option parsing, and decoded-size enforcement.
modules/ingest-attachment/build.gradle	Adds `ingest-common` to cluster modules for YAML REST tests.
docs/reference/enrich-processor/attachment.md	Documents the new `max_attachment_bytes` processor option.
docs/changelog/148493.yaml	Adds changelog entry for processor option + node setting.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.

marciw

Small edits to the associated docs changes.

Co-authored-by: Marci W <333176+marciw@users.noreply.github.com>

…ents-limit

marciw

~~LGTM, thanks!~~ sorry, I missed a couple of key things in the docs. Dismissing my review for now and will have more comments soon...

missed a couple of things

kingherc · 2026-05-18T07:13:21Z

Gentle reminder for reviews. Actually I just realized that for our sake, I could just introduce the node setting and no processor setting. However, since I've developed it, I leave it in here and welcome your comments on whether it'd be useful or not to introduce; else I can remove the per-processor setting. cc @jimczi , @tballison , @masseyke

tballison

LGTM

marciw

a few more docs tweaks. Am also going to push a commit to update the elasticsearch settings reference.

marciw · 2026-05-19T00:09:05Z

Am also going to push a commit to update the elasticsearch settings reference.

Nope, I'll do that in a separate PR so I can unblock you. 👍

Co-authored-by: Marci W <333176+marciw@users.noreply.github.com>

…ents-limit

kingherc · 2026-05-19T08:56:56Z

Thank you all! I have the necessary approvals to merge this. So, any last reviewer that would like to pitch in comments or feedback, please do so quickly. Even if it's just to tell me to wait before merging these.

Am also going to push a commit to update the elasticsearch settings reference.
Nope, I'll do that in a separate PR so I can unblock you. 👍

Thanks @marciw , may I ask where (in which repos and files) you do that? Just for my knowledge, because I have in my PR description the following:

Optional TODOs after this PR is merged:

extend Processors.ts in elastic/elasticsearch-specification if it is not done automatically.

see about AttachmentProcessor.java in elastic/elasticsearch-java

And I'm unaware whether I should do them, or they'd be automatically done by someone/something retroactively.

…ents-limit

marciw · 2026-05-19T22:07:00Z

Am also going to push a commit to update the elasticsearch settings reference.
Nope, I'll do that in a separate PR so I can unblock you. 👍

Thanks @marciw , may I ask where (in which repos and files) you do that?

just updating the configuration reference

Optional TODOs after this PR is merged:

extend Processors.ts in elastic/elasticsearch-specification if it is not done automatically.

see about AttachmentProcessor.java in elastic/elasticsearch-java

And I'm unaware whether I should do them, or they'd be automatically done by someone/something retroactively.

The first one (extending Processors.ts) is a task for you :) and the spec needs to be updated before the java client is regenerated.

For the second, I don't know all the details, but here's a recent java client PR for reference: elastic/elasticsearch-java#1236

…ents-limit

kingherc · 2026-05-21T07:09:49Z

@marciw This is merged, feel free to update the elasticsearch settings reference. Note that I'd recommend not documenting the ingest.attachment.max_field_size_message_suffix one since it's intended for integrating with Serverless.

Merged with elastic/elasticsearch#148493

kingherc self-assigned this May 7, 2026

kingherc added >enhancement :Distributed/Ingest Node Execution or management of Ingest Pipelines Team:Distributed Meta label for distributed team. v9.5.0 labels May 7, 2026

kingherc force-pushed the enhancement/attachments-limit branch from 3609f6f to 995276e Compare May 7, 2026 11:34

kingherc requested a review from Copilot May 7, 2026 11:59

Copilot started reviewing on behalf of kingherc May 7, 2026 12:00 View session

kingherc force-pushed the enhancement/attachments-limit branch from 995276e to 0170f78 Compare May 7, 2026 12:04

Copilot AI reviewed May 7, 2026

View reviewed changes

kingherc force-pushed the enhancement/attachments-limit branch from 50c018a to 51fbfe0 Compare May 7, 2026 13:06

kingherc requested a review from Copilot May 7, 2026 13:39

Copilot started reviewing on behalf of kingherc May 7, 2026 13:40 View session

Copilot AI reviewed May 7, 2026

View reviewed changes

kingherc force-pushed the enhancement/attachments-limit branch 2 times, most recently from 9ec81f8 to 75cd22e Compare May 7, 2026 14:22

kingherc requested a review from Copilot May 7, 2026 14:33

Copilot started reviewing on behalf of kingherc May 7, 2026 14:35 View session

Copilot AI reviewed May 7, 2026

View reviewed changes

Comment thread ...ingest-attachment/src/main/java/org/elasticsearch/ingest/attachment/AttachmentProcessor.java Outdated

kingherc force-pushed the enhancement/attachments-limit branch 2 times, most recently from d8b288d to 738e19f Compare May 8, 2026 09:45

kingherc requested a review from Copilot May 8, 2026 09:45

Copilot started reviewing on behalf of kingherc May 8, 2026 09:46 View session

kingherc force-pushed the enhancement/attachments-limit branch from 738e19f to 24582b7 Compare May 8, 2026 09:49

Copilot AI reviewed May 8, 2026

View reviewed changes

kingherc force-pushed the enhancement/attachments-limit branch from 24582b7 to c44c883 Compare May 8, 2026 12:58

marciw reviewed May 15, 2026

View reviewed changes

kingherc and others added 3 commits May 15, 2026 17:37

Update docs/reference/enrich-processor/attachment.md

3284992

Co-authored-by: Marci W <333176+marciw@users.noreply.github.com>

Apply suggestions from code review

a0f2573

Co-authored-by: Marci W <333176+marciw@users.noreply.github.com>

Merge remote-tracking branch 'kingherc/main' into enhancement/attachm…

408b915

…ents-limit

kingherc requested review from marciw and masseyke May 15, 2026 14:40

marciw previously approved these changes May 15, 2026

View reviewed changes

marciw self-requested a review May 15, 2026 17:37

tballison approved these changes May 18, 2026

View reviewed changes

marciw reviewed May 18, 2026

View reviewed changes

Comment thread docs/reference/enrich-processor/attachment.md Outdated

Comment thread docs/reference/enrich-processor/attachment.md

Comment thread docs/reference/enrich-processor/attachment.md Outdated

marciw approved these changes May 19, 2026

View reviewed changes

kingherc and others added 3 commits May 19, 2026 11:52

Apply suggestions from code review

81086ee

Co-authored-by: Marci W <333176+marciw@users.noreply.github.com>

PR comments

00e4ac1

Merge remote-tracking branch 'kingherc/main' into enhancement/attachm…

18b7a72

…ents-limit

Merge remote-tracking branch 'kingherc/main' into enhancement/attachm…

e5c9463

…ents-limit

marciw mentioned this pull request May 19, 2026

add attachment setting to es config reference #149439

Draft

kingherc added 4 commits May 20, 2026 10:20

Merge remote-tracking branch 'kingherc/main' into enhancement/attachm…

84aaf8b

…ents-limit

Merge remote-tracking branch 'kingherc/main' into enhancement/attachm…

2d3bfa0

…ents-limit

Merge remote-tracking branch 'kingherc/main' into enhancement/attachm…

2b5ab7f

…ents-limit

Merge remote-tracking branch 'kingherc/main' into enhancement/attachm…

bc27633

…ents-limit

kingherc merged commit 3a45b6a into elastic:main May 21, 2026
37 checks passed

kingherc deleted the enhancement/attachments-limit branch May 21, 2026 07:54

kingherc mentioned this pull request May 21, 2026

Add max_field_bytes attachment processor setting elastic/elasticsearch-specification#6300

Merged

kingherc added a commit to elastic/elasticsearch-specification that referenced this pull request May 21, 2026

Add max_field_bytes attachment processor setting (#6300)

fd94137

Merged with elastic/elasticsearch#148493

Conversation

kingherc commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented May 7, 2026

Uh oh!

github-actions Bot commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

github-actions Bot commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Vale Linting Results

Uh oh!

github-actions Bot commented May 7, 2026

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marciw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marciw left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kingherc commented May 18, 2026

Uh oh!

tballison left a comment

Choose a reason for hiding this comment

Uh oh!

marciw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marciw commented May 19, 2026

Uh oh!

kingherc commented May 19, 2026

Uh oh!

marciw commented May 19, 2026

Uh oh!

kingherc commented May 7, 2026 •

edited

Loading

github-actions Bot commented May 7, 2026 •

edited

Loading

github-actions Bot commented May 7, 2026 •

edited

Loading

marciw left a comment •

edited

Loading