Repository navigation
[Inference API] Add multimodal image rerank support for the Elastic Inference Service - #152012
Conversation
Allow DataType.IMAGE in the rerank task's supported data types and have the Elastic Inference Service accept image inputs and queries by overriding supportsMultimodalRerank. Update and add tests covering image parsing, request validation, and the request sent to EIS. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
🔍 Preview links for changed docs⏳ Building and deploying preview... View progress This comment will be updated with preview links when the build is complete. |
|
Pinging @elastic/search-inference-team (Team:Search - Inference) |
|
@coderabbitai full review |
ℹ️ Important: Docs version tagging👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version. We use applies_to tags to mark version-specific features and changes. Expand for a quick overviewWhen to use applies_to tags:✅ At the page level to indicate which products/deployments the content applies to (mandatory) What NOT to do:❌ Don't remove or replace information that applies to an older version 🤔 Need help?
|
✅ Action performedFull review finished. |
WalkthroughRerank parsing and validation now use separate allowlists for inputs and queries, with IMAGE allowed for rerank inputs. ElasticInferenceService now reports multimodal rerank support. Tests and the changelog were updated for image and mixed-input rerank flows. ChangesMultimodal Rerank Support
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@Jan-Kazlouski-elastic is it possible to test this end to end with a rerank model hosted in the Elastic Inference Service? Is it available in QA? If so, let's add |
It is not yet possible to to test this end to end with EIS.
According to the https://github.com/elastic/search-team/issues/14266,
As mentioned above - it is not yet available in QA, so E2E testing is not yet possible, but I'm looking forward to you showing me a way to test it. |
…tests - Updated RerankRequest to specify that the "query" field only supports DataType.TEXT. - Refactored parsing methods to utilize specific supported data types for inputs and queries. - Modified related tests to reflect the new query type constraints and ensure proper validation for unsupported data types. - Adjusted imports in various test classes to align with the new data type constants.
|
@coderabbitai full review |
✅ Action performedFull review finished. |
jonathan-buttner
left a comment
There was a problem hiding this comment.
Looking good, left some feedback
Restrict rerank queries to text in the service layer so non-text queries are rejected even for services that support image inputs, and update supportsMultimodalRerank docs accordingly. Restore the Elastic Inference Service tests asserting non-text queries throw. Rename randomInferenceString to randomTextInferenceString since it is used for both the query and input fields. Add a parser test for an object text query with a text+image input array.
Updated the SenderService and ServiceUtils to introduce specific exceptions for unsupported multimodal rerank queries and inputs. The changes include renaming the existing exception method and adding new methods to handle different cases. Adjusted the rerankInfer method in various services to utilize the new exception handling logic, ensuring clearer error messages for non-text inputs and queries. Updated related tests to reflect these changes.
Allow both text and image for the rerank query and input fields by using a single SUPPORTED_RERANK_DATA_TYPES set, instead of restricting the query to text. The Elastic Inference Service accepts non-text queries and inputs via supportsMultimodalRerank, and the per-field exception split is reverted to the shared unsupported-rerank error. Update parser, validation, and service tests accordingly.
…nference Service (elastic#152012) * Add multimodal image rerank support for EIS Allow DataType.IMAGE in the rerank task's supported data types and have the Elastic Inference Service accept image inputs and queries by overriding supportsMultimodalRerank. Update and add tests covering image parsing, request validation, and the request sent to EIS. Co-authored-by: Cursor <cursoragent@cursor.com> * Add changelog for EIS multimodal rerank Co-authored-by: Cursor <cursoragent@cursor.com> * Refactor RerankRequest to enforce query type restrictions and update tests - Updated RerankRequest to specify that the "query" field only supports DataType.TEXT. - Refactored parsing methods to utilize specific supported data types for inputs and queries. - Modified related tests to reflect the new query type constraints and ensure proper validation for unsupported data types. - Adjusted imports in various test classes to align with the new data type constants. * Address review feedback for multimodal rerank Restrict rerank queries to text in the service layer so non-text queries are rejected even for services that support image inputs, and update supportsMultimodalRerank docs accordingly. Restore the Elastic Inference Service tests asserting non-text queries throw. Rename randomInferenceString to randomTextInferenceString since it is used for both the query and input fields. Add a parser test for an object text query with a text+image input array. * Refactor multimodal rerank exception handling Updated the SenderService and ServiceUtils to introduce specific exceptions for unsupported multimodal rerank queries and inputs. The changes include renaming the existing exception method and adding new methods to handle different cases. Adjusted the rerankInfer method in various services to utilize the new exception handling logic, ensuring clearer error messages for non-text inputs and queries. Updated related tests to reflect these changes. * Unify rerank query and input data types Allow both text and image for the rerank query and input fields by using a single SUPPORTED_RERANK_DATA_TYPES set, instead of restricting the query to text. The Elastic Inference Service accepts non-text queries and inputs via supportsMultimodalRerank, and the per-field exception split is reverted to the shared unsupported-rerank error. Update parser, validation, and service tests accordingly. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
Closes https://github.com/elastic/search-team/issues/14255
Summary
DataType.IMAGEtoRerankRequest.SUPPORTED_RERANK_DATA_TYPESso image inputs are accepted by the rerank task parser and validation.supportsMultimodalRerank()inElasticInferenceServiceto returntrue, allowing theelasticservice to handle multimodal (image) rerank requests. Action-level validation still restricts rerank inputs to text and image, so only images (not audio/video/pdf) can reach EIS.InferenceStringobjects (including images) correctly, so no production change was needed there.Is it safe?
Enabling multimodal rerank inputs for the
elasticintegration now, ahead of a multimodal-capable rerank model (m0) being added to EIS, is safe: the only thing that changes is where an invalid-input error originates. Without a model that accepts non-text inputs, EIS will simply reject the inputs as invalid for the specified model, rather than the inference API rejecting them up front — the same behavior a user already sees today if they send non-text inputs to any rerank model other than m0.Doing it in this order also has a benefit: once m0 lands in EIS, synthetic end-to-end tests can be written against it immediately, instead of waiting for Elasticsearch to be updated first.
Note: the user-facing documentation update is intentionally left as the last step, so we only advertise the feature once it's verified working end to end.
Tests
RerankRequestTests: added parser tests for image inputand mixed text+image inputs; updated the unsupported-type assertions to reflect the new supported set[text, image].RerankActionRequestTests: added validation tests confirming image inputspass validation.ElasticInferenceServiceTests: addedtestRerankInfer_SendsMultimodalRerankRequestverifying the outgoing request preserves imagetype/format/valueobjects; removed the now-obsolete tests that asserted non-text rerank inputs are rejected by theelasticservice.Testing
Endpoint:
POST /_inference/rerank/{inference_id}?timeout=5mModels:
jina-reranker-m0(multimodal) ·jina-reranker-v3(text-only, test 12)Image placeholder — replace
<IMG>in requests with:1 — Text-only, string array · 200
RQ
{ "query": "golden retriever playing in a park", "input": [ "A dog runs through green grass with a tennis ball.", "A cat sleeps on a windowsill.", "A golden retriever catches a frisbee outdoors." ] }RS
{ "rerank": [ { "index": 2, "relevance_score": 0.960489 }, { "index": 0, "relevance_score": 0.85283023 }, { "index": 1, "relevance_score": 0.5741337 } ] }2 — Mixed text + image, string query · 200
RQ
{ "query": "golden retriever playing in a park", "input": [ { "type": "text", "value": "A cat sleeps on a windowsill." }, { "type": "image", "value": "<IMG>" } ] }RS
{ "rerank": [ { "index": 0, "relevance_score": 0.5788402 }, { "index": 1, "relevance_score": 0.44391167 } ] }3 — Mixed inputs, explicit formats · 200
RQ
{ "query": { "type": "text", "format": "text", "value": "golden retriever playing in a park" }, "input": [ { "type": "text", "format": "text", "value": "A golden retriever catches a frisbee outdoors." }, { "type": "image", "format": "base64", "value": "<IMG>" }, { "type": "text", "format": "text", "value": "A dog runs through green grass with a tennis ball." } ] }RS
{ "rerank": [ { "index": 0, "relevance_score": 0.96133167 }, { "index": 2, "relevance_score": 0.85283023 }, { "index": 1, "relevance_score": 0.44938895 } ] }4 — Single image input (object, not array) · 200
RQ
{ "query": "dog playing fetch in a park", "input": { "type": "image", "format": "base64", "value": "<IMG>" } }RS
{ "rerank": [ { "index": 0, "relevance_score": 0.48258406 } ] }5 — Image query, text candidates · 200
RQ
{ "query": { "type": "image", "format": "base64", "value": "<IMG>" }, "input": [ "A golden retriever catches a frisbee outdoors.", "A cat sleeps on a windowsill.", "A dog runs through green grass with a tennis ball." ] }RS
{ "rerank": [ { "index": 2, "relevance_score": 0.4601458 }, { "index": 1, "relevance_score": 0.4465505 }, { "index": 0, "relevance_score": 0.42868888 } ] }6 — Full multimodal, top_n=2 · 200
RQ
{ "query": { "type": "image", "format": "base64", "value": "<IMG>" }, "input": [ { "type": "text", "value": "A cat on a sofa." }, { "type": "image", "format": "base64", "value": "<IMG>" }, { "type": "text", "value": "A golden retriever in a park." } ], "top_n": 2 }RS
{ "rerank": [ { "index": 1, "relevance_score": 0.96717536 }, { "index": 0, "relevance_score": 0.45577812 } ] }7 — String array, top_n=1 · 200
RQ
{ "query": "Which passage describes outdoor dog activity?", "input": [ "Indoor cooking recipe for pasta.", "A dog runs through green grass with a tennis ball.", "Stock market analysis for Q3." ], "top_n": 1 }RS
{ "rerank": [ { "index": 1, "relevance_score": 0.8200929 } ] }8 — return_documents not supported · 400
RQ
{ "query": "golden retriever playing in a park", "input": [ { "type": "text", "value": "A dog runs through green grass with a tennis ball." }, { "type": "image", "format": "base64", "value": "<IMG>" } ], "top_n": 2, "return_documents": true }RS
{ "error": { "type": "validation_exception", "reason": "Validation Failed: 1: Invalid return_documents [true]. The return_documents option is not supported by this service;" }, "status": 400 }9 — Empty input array · 400
RQ
{ "query": "test query", "input": [] }RS
{ "error": { "type": "action_request_validation_exception", "reason": "Validation Failed: 1: Field [input] cannot be an empty array;" }, "status": 400 }10 — Invalid top_n=0 · 400
RQ
{ "query": "test query", "input": ["doc one", "doc two"], "top_n": 0 }RS
{ "error": { "type": "action_request_validation_exception", "reason": "Validation Failed: 1: Field [top_n] must be greater than or equal to 1;" }, "status": 400 }11 — Unsupported input type (audio) · 400
RQ
{ "query": "test query", "input": [ { "type": "audio", "format": "base64", "value": "data:audio/wav;base64,abcd" } ] }RS
{ "error": { "type": "x_content_parse_exception", "reason": "[4:82] [RerankRequest] failed to parse field [input]", "caused_by": { "type": "x_content_parse_exception", "reason": "Field [input] contains unsupported [type] value [audio]. Supported values are [text, image]" } }, "status": 400 }12 — Text-only model rejects image · 400 ·
jina-reranker-v3RQ
{ "query": "test", "input": [ { "type": "image", "format": "base64", "value": "<IMG>" } ] }RS
{ "error": { "type": "status_exception", "reason": "Received a bad request status code for request from inference entity id [%inference entity id%] status [400]. Error message: [validation failed: field [documents[0]] must be a content type supported by model jina-reranker-v3, but got \"image\" (allowed: [text])]" }, "status": 400 }