Skip to content

[Inference API] Add multimodal image rerank support for the Elastic Inference Service - #152012

Merged
jonathan-buttner merged 17 commits into
elastic:mainfrom
Jan-Kazlouski-elastic:eis-multimodal-rerank
Jul 6, 2026
Merged

jonathan-buttner merged 17 commits into
elastic:mainfrom
Jan-Kazlouski-elastic:eis-multimodal-rerank

Conversation

@Jan-Kazlouski-elastic

@Jan-Kazlouski-elastic Jan-Kazlouski-elastic commented Jun 23, 2026 •

Copy link
Copy Markdown
Contributor

Closes https://github.com/elastic/search-team/issues/14255

Summary

  • Add DataType.IMAGE to RerankRequest.SUPPORTED_RERANK_DATA_TYPES so image inputs are accepted by the rerank task parser and validation.
  • Override supportsMultimodalRerank() in ElasticInferenceService to return true, allowing the elastic service to handle multimodal (image) rerank requests. Action-level validation still restricts rerank inputs to text and image, so only images (not audio/video/pdf) can reach EIS.
  • The EIS request entity already serializes InferenceString objects (including images) correctly, so no production change was needed there.

Is it safe?

Enabling multimodal rerank inputs for the elastic integration now, ahead of a multimodal-capable rerank model (m0) being added to EIS, is safe: the only thing that changes is where an invalid-input error originates. Without a model that accepts non-text inputs, EIS will simply reject the inputs as invalid for the specified model, rather than the inference API rejecting them up front — the same behavior a user already sees today if they send non-text inputs to any rerank model other than m0.

Doing it in this order also has a benefit: once m0 lands in EIS, synthetic end-to-end tests can be written against it immediately, instead of waiting for Elasticsearch to be updated first.

Note: the user-facing documentation update is intentionally left as the last step, so we only advertise the feature once it's verified working end to end.

Tests

  • RerankRequestTests: added parser tests for image inputand mixed text+image inputs; updated the unsupported-type assertions to reflect the new supported set [text, image].
  • RerankActionRequestTests: added validation tests confirming image inputspass validation.
  • ElasticInferenceServiceTests: added testRerankInfer_SendsMultimodalRerankRequest verifying the outgoing request preserves image type/format/value objects; removed the now-obsolete tests that asserted non-text rerank inputs are rejected by the elastic service.

Testing

Endpoint: POST /_inference/rerank/{inference_id}?timeout=5m

Models: jina-reranker-m0 (multimodal) · jina-reranker-v3 (text-only, test 12)

Image placeholder — replace <IMG> in requests with:

data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAMwAAADACAMAAAB/Pny7AAAA7VBMVEX///8AAABONC780K49Wv5gfYu8vLwiIiIAvNRHLypceJ5hfoc4Vf//1bL8/PxSbsCCgoLk5OQpKSlOQDXctpgZEA9AXv8SG0sGCRorHRocKnY4U+sKDQ7rwqISGBssOkE+Pj5fX19MY29ZdIF1YFGHcF68m4EjLTKSkpInOqIcJSndzbU9UFlcv87DyrvrzrF1wcpOTk6jo6OixsE7MCg4JSHLy8skNZLNqo4EBQ9kU0VZSj0uJh93d3cyMjKihnBvamZca3KoqbI8R5YaLI41R3omM1lNZ7EAAEEbIy46TGcwPk8jEQyIw8eZjobFTeMIAAAFHUlEQVR4nO3da0PaOhwG8CGOHqYwKqBjFKQ6sJt63Biy6Siw+/18/48zSP7FhqU5XNr04vP4igRCfmsX2jSFBw+2TTm0bN2V7ePkQooTt2SWvhGOxejHLZml3w4H0wYm5ACTWExIA0A8GNN+5c/YYn2pF7dNh7dX0YvpyP5hG8WdLdPgDdnAAANM6jD1dGMa10K2tXiYTp9HzxmBh9l6U8gxlI4JDDDAABNRyibLsFNnCRtzzZutc8x4yN8tqhG6cGDNQ4qwLV6KtGnYe1kHhagwRkif9StheAxggAEGmJRidmiyhj5vDjosoc+qa8JQ6sIWCn0CSiumCAwwwNxfzA5N+tQzgaE0gAEGGGBCU5hDFmfUYNFpCR/jjFkGWjdJVJgKb1DvJgEGGGCAiQXjzeEXpaVi6GJuUVrppRgrRnZ4cJ2TpeFhpLU5oaFYMEU5xgIGGGDuDybXEMMLB5Meyy11VKgcUSVlwkstek7oszPrYKS5bZVYurLKwduSPzVpCwnCvKuV8vMEYfJ3AQaYLGBc3uCvjTHVBGEKlXmcqWoBoxxT7bJMWry/va4kk5qIoeJRRBi6japg5IJXAMkx3RbLoqstWfJieGGtGhGGopwEDMDkS/mNUmolEbNpgAEmuxi+OoTmAKxB1Z8Jde2KR97vK1ktYSy6RUjTchNxaeWoV/OHht3z35fzvPxXannNKi/FSsIYfb5UM/Tlp3KMuOh1UBOO52lgPr/8h0WOeckrX0sxelc1/YWR9BcYYO43ZkeBGaUM482biHNB72hypZUujBcR86wlDMapx8h6CgwwwGQTQ3M12cCIVytSjskBAwww/4ORXqBMKWZo80hNSszVb9mchbIyaox3B+14bUz+6pxFPtd0LquMGkORf+2EGrN+gAEGmIRijANf2qnGlIcFf1wrVIx3gfbZSAtmKfRlbeFhhL1XN6YNDDDRY7L0f8ZZDM3B07MB/ZZmae2MXszQYStr/lNNnMstrZ4stKzRqPAMtWI8Ez8ukF/SCNihxLU+YjR9vZESI7/YFIAZAAMMMMuLGlRRYsZxYkyXzdxMxeUmyvSmdnCmcWJo6sZ0qyvHNVVJwJfRl23FrrMUOwH9Vcacro6JdU9aJcAkNaa9OsZOOqbssrvtO3T1oz4a+DKi5YJGhz3JTfoAQFM3Q9rbbsXDe7qzaUpPSjrGC52ydcXPfLqxIQk/AbJOPIx4OAZM/AEmqcniACAfmlOKkQeYGANMUgNMjFFORzjts8C0HeVLY8HYwkVnMcbJQ0VOVK/U+ysnC4xqT7pQYS5UrwQGGGASjaHfJbVz7XlokaPV9sdSj2ZLT/a3MMPo/N1Ts+KyS6fvT1iOeV/OToScqjCn4nPPuOWYP3rPGncrmn6yhdZoUn8vOOZY2X0l7ZhjaM885a1ruj7jrTeLFqP5x3SAASaS8CFzhrmZJToMa32GiXSENvk6xg8fP72Z5dNjns83rC9fvj7eMF+/sAZuPtNj3vrHD/zdotpABb4DfGsesuzuz7P7/Akrfdrkj9fObvMpa+DJc2qQt978xt8t4ltOjpq7vhzeYTbMAnMolB6x0qjvnwEGGGCAAQYYYIABJjmY74+E/ODnMz8fbZyfrAHrh1j6XQvmxemeP4uTs70Nszg5E0tfaMIIJ4phn2l6pcAAAwwwwAADDDBRYvYWfz6Mr3Bv6U9V4MP46jVhMnXUfCTMkN9NnG82b76/vzRx7rWLkzNggAEGmCxg/gAcTwKRD+vGjgAAAABJRU5ErkJggg==

1 — Text-only, string array · 200

RQ

{
  "query": "golden retriever playing in a park",
  "input": [
    "A dog runs through green grass with a tennis ball.",
    "A cat sleeps on a windowsill.",
    "A golden retriever catches a frisbee outdoors."
  ]
}

RS

{
  "rerank": [
    { "index": 2, "relevance_score": 0.960489 },
    { "index": 0, "relevance_score": 0.85283023 },
    { "index": 1, "relevance_score": 0.5741337 }
  ]
}
2 — Mixed text + image, string query · 200

RQ

{
  "query": "golden retriever playing in a park",
  "input": [
    { "type": "text", "value": "A cat sleeps on a windowsill." },
    { "type": "image", "value": "<IMG>" }
  ]
}

RS

{
  "rerank": [
    { "index": 0, "relevance_score": 0.5788402 },
    { "index": 1, "relevance_score": 0.44391167 }
  ]
}
3 — Mixed inputs, explicit formats · 200

RQ

{
  "query": { "type": "text", "format": "text", "value": "golden retriever playing in a park" },
  "input": [
    { "type": "text", "format": "text", "value": "A golden retriever catches a frisbee outdoors." },
    { "type": "image", "format": "base64", "value": "<IMG>" },
    { "type": "text", "format": "text", "value": "A dog runs through green grass with a tennis ball." }
  ]
}

RS

{
  "rerank": [
    { "index": 0, "relevance_score": 0.96133167 },
    { "index": 2, "relevance_score": 0.85283023 },
    { "index": 1, "relevance_score": 0.44938895 }
  ]
}
4 — Single image input (object, not array) · 200

RQ

{
  "query": "dog playing fetch in a park",
  "input": {
    "type": "image",
    "format": "base64",
    "value": "<IMG>"
  }
}

RS

{
  "rerank": [
    { "index": 0, "relevance_score": 0.48258406 }
  ]
}
5 — Image query, text candidates · 200

RQ

{
  "query": { "type": "image", "format": "base64", "value": "<IMG>" },
  "input": [
    "A golden retriever catches a frisbee outdoors.",
    "A cat sleeps on a windowsill.",
    "A dog runs through green grass with a tennis ball."
  ]
}

RS

{
  "rerank": [
    { "index": 2, "relevance_score": 0.4601458 },
    { "index": 1, "relevance_score": 0.4465505 },
    { "index": 0, "relevance_score": 0.42868888 }
  ]
}
6 — Full multimodal, top_n=2 · 200

RQ

{
  "query": { "type": "image", "format": "base64", "value": "<IMG>" },
  "input": [
    { "type": "text", "value": "A cat on a sofa." },
    { "type": "image", "format": "base64", "value": "<IMG>" },
    { "type": "text", "value": "A golden retriever in a park." }
  ],
  "top_n": 2
}

RS

{
  "rerank": [
    { "index": 1, "relevance_score": 0.96717536 },
    { "index": 0, "relevance_score": 0.45577812 }
  ]
}
7 — String array, top_n=1 · 200

RQ

{
  "query": "Which passage describes outdoor dog activity?",
  "input": [
    "Indoor cooking recipe for pasta.",
    "A dog runs through green grass with a tennis ball.",
    "Stock market analysis for Q3."
  ],
  "top_n": 1
}

RS

{
  "rerank": [
    { "index": 1, "relevance_score": 0.8200929 }
  ]
}
8 — return_documents not supported · 400

RQ

{
  "query": "golden retriever playing in a park",
  "input": [
    { "type": "text", "value": "A dog runs through green grass with a tennis ball." },
    { "type": "image", "format": "base64", "value": "<IMG>" }
  ],
  "top_n": 2,
  "return_documents": true
}

RS

{
  "error": {
    "type": "validation_exception",
    "reason": "Validation Failed: 1: Invalid return_documents [true]. The return_documents option is not supported by this service;"
  },
  "status": 400
}
9 — Empty input array · 400

RQ

{
  "query": "test query",
  "input": []
}

RS

{
  "error": {
    "type": "action_request_validation_exception",
    "reason": "Validation Failed: 1: Field [input] cannot be an empty array;"
  },
  "status": 400
}
10 — Invalid top_n=0 · 400

RQ

{
  "query": "test query",
  "input": ["doc one", "doc two"],
  "top_n": 0
}

RS

{
  "error": {
    "type": "action_request_validation_exception",
    "reason": "Validation Failed: 1: Field [top_n] must be greater than or equal to 1;"
  },
  "status": 400
}
11 — Unsupported input type (audio) · 400

RQ

{
  "query": "test query",
  "input": [
    { "type": "audio", "format": "base64", "value": "data:audio/wav;base64,abcd" }
  ]
}

RS

{
  "error": {
    "type": "x_content_parse_exception",
    "reason": "[4:82] [RerankRequest] failed to parse field [input]",
    "caused_by": {
      "type": "x_content_parse_exception",
      "reason": "Field [input] contains unsupported [type] value [audio]. Supported values are [text, image]"
    }
  },
  "status": 400
}
12 — Text-only model rejects image · 400 · jina-reranker-v3

RQ

{
  "query": "test",
  "input": [
    { "type": "image", "format": "base64", "value": "<IMG>" }
  ]
}

RS

{
  "error": {
    "type": "status_exception",
    "reason": "Received a bad request status code for request from inference entity id [%inference entity id%] status [400]. Error message: [validation failed: field [documents[0]] must be a content type supported by model jina-reranker-v3, but got \"image\" (allowed: [text])]"
  },
  "status": 400
}

Allow DataType.IMAGE in the rerank task's supported data types and
have the Elastic Inference Service accept image inputs and queries by
overriding supportsMultimodalRerank. Update and add tests covering
image parsing, request validation, and the request sent to EIS.

Co-authored-by: Cursor <cursoragent@cursor.com>
@elasticsearchmachine elasticsearchmachine added external-contributor Pull request authored by a developer outside the Elasticsearch team v9.5.0 needs:triage Requires assignment of a team area label labels Jun 23, 2026
Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Jun 23, 2026 •

Copy link
Copy Markdown
Contributor

🔍 Preview links for changed docs

⏳ Building and deploying preview... View progress

This comment will be updated with preview links when the build is complete.

@Jan-Kazlouski-elastic
Jan-Kazlouski-elastic requested a review from a team June 23, 2026 16:52
@elasticsearchmachine elasticsearchmachine removed the needs:triage Requires assignment of a team area label label Jun 23, 2026
@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Pinging @elastic/search-inference-team (Team:Search - Inference)

@Jan-Kazlouski-elastic

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

@github-actions

Copy link
Copy Markdown
Contributor

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

@coderabbitai

coderabbitai Bot commented Jun 23, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Jun 23, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Rerank parsing and validation now use separate allowlists for inputs and queries, with IMAGE allowed for rerank inputs. ElasticInferenceService now reports multimodal rerank support. Tests and the changelog were updated for image and mixed-input rerank flows.

Changes

Multimodal Rerank Support

Layer / File(s) Summary
Request contract and validation
server/src/main/java/org/elasticsearch/inference/RerankRequest.java, x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/inference/action/RerankAction.java, docs/changelog/152012.yaml
RerankRequest splits input and query data-type allowlists, routes parsing through the matching allowlist for each field, and RerankAction.Request.validate() checks rerank inputs and query against separate allowlists. The changelog adds the inference entry for PR 152012.
Parser and validation tests
server/src/test/java/org/elasticsearch/inference/RerankRequestTests.java, x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/inference/action/RerankActionRequestTests.java, x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/external/http/sender/QueryAndDocsInputsTests.java
Parser coverage adds image-input and mixed-input cases, unsupported-type assertions use the input/query-specific allowlists, random generation helpers split into input/query variants, and related request-construction tests use the matching supported type sets.
Service capability and HTTP integration
x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/elastic/ElasticInferenceService.java, x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/elastic/ElasticInferenceServiceTests.java
ElasticInferenceService overrides supportsMultimodalRerank() to return true, and the integration test adds a multimodal rerank request case while removing obsolete non-text validation tests.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested reviewers

  • jonathan-buttner
  • timgrein

Poem

🐇 A carrot-bright rerank took flight,
With image inputs joining text just right.
The query stayed tidy, text-only and neat,
بينما the tests hopped along in sync so sweet.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 2.86% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: adding multimodal image rerank support for Elastic Inference Service.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The description matches the changeset by describing multimodal rerank support, validation updates, service capability changes, and related tests.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@jonathan-buttner

Copy link
Copy Markdown
Contributor

@Jan-Kazlouski-elastic is it possible to test this end to end with a rerank model hosted in the Elastic Inference Service? Is it available in QA? If so, let's add cloud-deploy and I can show you how to deploy a ES cluster to test with.

@Jan-Kazlouski-elastic

Copy link
Copy Markdown
Contributor Author

Hi @jonathan-buttner

Is it possible to test this end to end with a rerank model hosted in the Elastic Inference Service?

It is not yet possible to to test this end to end with EIS.

Is it available in QA?

According to the https://github.com/elastic/search-team/issues/14266, jina-reranker-m0 model is yet to be added to QA env.

If so, let's add cloud-deploy and I can show you how to deploy a ES cluster to test with.

As mentioned above - it is not yet available in QA, so E2E testing is not yet possible, but I'm looking forward to you showing me a way to test it.

@Jan-Kazlouski-elastic Jan-Kazlouski-elastic changed the title Add multimodal image rerank support for the Elastic Inference Service [Inference API] Add multimodal image rerank support for the Elastic Inference Service Jun 26, 2026
…tests

- Updated RerankRequest to specify that the "query" field only supports DataType.TEXT.
- Refactored parsing methods to utilize specific supported data types for inputs and queries.
- Modified related tests to reflect the new query type constraints and ensure proper validation for unsupported data types.
- Adjusted imports in various test classes to align with the new data type constants.
@Jan-Kazlouski-elastic

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Jun 26, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@jonathan-buttner jonathan-buttner left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, left some feedback

@jonathan-buttner jonathan-buttner added the cloud-deploy Publish cloud docker image for Cloud-First-Testing label Jun 26, 2026
Restrict rerank queries to text in the service layer so non-text
queries are rejected even for services that support image inputs,
and update supportsMultimodalRerank docs accordingly. Restore the
Elastic Inference Service tests asserting non-text queries throw.

Rename randomInferenceString to randomTextInferenceString since it
is used for both the query and input fields. Add a parser test for
an object text query with a text+image input array.
Updated the SenderService and ServiceUtils to introduce specific exceptions for unsupported multimodal rerank queries and inputs. The changes include renaming the existing exception method and adding new methods to handle different cases. Adjusted the rerankInfer method in various services to utilize the new exception handling logic, ensuring clearer error messages for non-text inputs and queries. Updated related tests to reflect these changes.
Allow both text and image for the rerank query and input fields by
using a single SUPPORTED_RERANK_DATA_TYPES set, instead of restricting
the query to text. The Elastic Inference Service accepts non-text
queries and inputs via supportsMultimodalRerank, and the per-field
exception split is reverted to the shared unsupported-rerank error.

Update parser, validation, and service tests accordingly.
@jonathan-buttner
jonathan-buttner enabled auto-merge (squash) July 6, 2026 14:18
@jonathan-buttner
jonathan-buttner merged commit 18d0151 into elastic:main Jul 6, 2026
39 checks passed
burqen pushed a commit to burqen/elasticsearch that referenced this pull request Jul 7, 2026
…nference Service (elastic#152012)

* Add multimodal image rerank support for EIS

Allow DataType.IMAGE in the rerank task's supported data types and
have the Elastic Inference Service accept image inputs and queries by
overriding supportsMultimodalRerank. Update and add tests covering
image parsing, request validation, and the request sent to EIS.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Add changelog for EIS multimodal rerank

Co-authored-by: Cursor <cursoragent@cursor.com>

* Refactor RerankRequest to enforce query type restrictions and update tests

- Updated RerankRequest to specify that the "query" field only supports DataType.TEXT.
- Refactored parsing methods to utilize specific supported data types for inputs and queries.
- Modified related tests to reflect the new query type constraints and ensure proper validation for unsupported data types.
- Adjusted imports in various test classes to align with the new data type constants.

* Address review feedback for multimodal rerank

Restrict rerank queries to text in the service layer so non-text
queries are rejected even for services that support image inputs,
and update supportsMultimodalRerank docs accordingly. Restore the
Elastic Inference Service tests asserting non-text queries throw.

Rename randomInferenceString to randomTextInferenceString since it
is used for both the query and input fields. Add a parser test for
an object text query with a text+image input array.

* Refactor multimodal rerank exception handling

Updated the SenderService and ServiceUtils to introduce specific exceptions for unsupported multimodal rerank queries and inputs. The changes include renaming the existing exception method and adding new methods to handle different cases. Adjusted the rerankInfer method in various services to utilize the new exception handling logic, ensuring clearer error messages for non-text inputs and queries. Updated related tests to reflect these changes.

* Unify rerank query and input data types

Allow both text and image for the rerank query and input fields by
using a single SUPPORTED_RERANK_DATA_TYPES set, instead of restricting
the query to text. The Elastic Inference Service accepts non-text
queries and inputs via supportsMultimodalRerank, and the per-field
exception split is reverted to the shared unsupported-rerank error.

Update parser, validation, and service tests accordingly.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cloud-deploy Publish cloud docker image for Cloud-First-Testing >enhancement external-contributor Pull request authored by a developer outside the Elasticsearch team :SearchOrg/Inference Label for the Search Inference team Team:Search - Inference v9.5.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants