Repository navigation
[Inference API] Task settings are ignored for all Amazon Bedrock and certain Google Vertex chat completions #148792
Description
Activity
Pinging @elastic/search-inference-team (Team:Search - Inference)
Opened #149268 with a fix. For the Bedrock side I added a topK field on the unified entity and routed it through the same additional-fields channel the non-unified path uses, so the wiring is consistent. For Vertex it's just passing the configured max_tokens through and falling back to it when the per-request value is null. Maintainers, let me know if anything needs adjusting.
The PR to fix this issue on main should be able to be backported to 9.4 cleanly, but for 9.3, only the Google Vertex changes can be backported, since Amazon Bedrock didn't support chat completion in 9.3. I'll take care of any backport merge conflicts once the main PR is merged.
- added a commit that references this issue
on May 19, 2026 - added a commit that references this issue
on May 20, 2026 - added a commit that references this issue
on May 20, 2026
When creating a
chat_completionendpoint using theamazonbedrockorgooglevertexaiservices, it is possible to specify atask_settingsfield containing some settings that are applicable tochat_completion. For Amazon Bedrock, the relevant settings aremax_new_tokens,temperature,top_pandtop_k. For Google Vertex, the relevant setting ismax_tokens.When task settings are specified at endpoint creation time, they should be used for all requests using that endpoint unless explicitly overridden in the request. However, when an
AmazonBedrockChatCompletionRequestEntityis created inAmazonBedrockChatCompletionEntityFactory, the values used formax_new_tokens,temperatureandtop_pare taken only from the request, with the task settings associated with the model being ignored, and thetop_ksetting being omitted entirely, since it can't be specified in a request (see the code here).Similarly, for a
GoogleVertexAiUnifiedChatCompletionRequestEntity(used when theproviderservice settings is omitted or set togoogle), the task settings are ignored, meaning that themax_tokenstask setting is not used (see the code here).To fix this bug, the values configured in the task settings should be used if the corresponding values in the request are null.
For Amazon Bedrock chat completions, this bug is present in 9.4 and onward (introduced in #139411, which added
chat_completionfor Amazon Bedrock).For Google Vertex, this bug is present in 9.2 and onward (introduced in #134080, which added the
max_tokenstask setting).