Skip to content

Inference API returning Unable to execute HTTP request: event executor terminated #117916

Closed
#118177
Closed
Inference API returning Unable to execute HTTP request: event executor terminated#117916
#118177
Assignees
ymao1
Labels
:mlMachine learning>bugFeature:GenAIFeatures around GenAITeam:MLMeta label for the ML team

Description

@jeffvestal
jeffvestal
opened on Dec 3, 2024

Elasticsearch Version

serverless

Installed Plugins

No response

Java Version

bundled

OS Version

serverless

Problem Description

Getting error

{
  "error": {
    "root_cause": [
      {
        "type": "exception",
        "reason": "Amazon Bedrock converse call failed"
      }
    ],
    "type": "exception",
    "reason": "Failed to send request from inference entity id [bedrock_cohere-cmd-r-plus_completion]",
    "caused_by": {
      "type": "exception",
      "reason": "org.elasticsearch.ElasticsearchException: Amazon Bedrock converse call failed",
      "caused_by": {
        "type": "exception",
        "reason": "Amazon Bedrock converse call failed",
        "caused_by": {
          "type": "sdk_client_exception",
          "reason": "Unable to execute HTTP request: event executor terminated",
          "caused_by": {
            "type": "rejected_execution_exception",
            "reason": "event executor terminated"
          }
        }
      }
    }
  },
  "status": 500
}

Steps to Reproduce

## Using Bedrock Service - Cohere Command R+ model
PUT _inference/completion/bedrock_cohere-cmd-r-plus_completion
{
    "service": "amazonbedrock",
    "service_settings": {
        "access_key": "${bedrock_access_key}",
        "secret_key": "${bedrock_secret_key}",
        "region": "us-east-1",
        "provider": "cohere",
        "model": "cohere.command-r-plus-v1:0"
    }
}

## Test inference api - streaming
POST _inference/completion/bedrock_cohere-cmd-r-plus_completion/
{
  "input": "How many male malard ducks fit in an american football field?"
}

Logs (if relevant)

No response

Activity

  1. jeffvestal
    added
    :SearchOrg/InferenceLabel for the Search Inference team
    >bug
    needs:triageRequires assignment of a team area label
    Team:MLMeta label for the ML team
    on Dec 3, 2024
  2. elasticsearchmachine
    added
    Team:SearchOrgMeta label for the Search Org (Enterprise Search)
    Team:Search - Inference
    and removed
    Team:MLMeta label for the ML team
    needs:triageRequires assignment of a team area label
    on Dec 3, 2024

  3. elasticsearchmachine commented on Dec 3, 2024

    @elasticsearchmachine
    elasticsearchmachine
    on Dec 3, 2024
    Collaborator

    Pinging @elastic/search-inference-team (Team:Search - Inference)

  4. elasticsearchmachine commented on Dec 3, 2024

    @elasticsearchmachine
    elasticsearchmachine
    on Dec 3, 2024
    Collaborator

    Pinging @elastic/search-eng (Team:SearchOrg)

  5. jeffvestal commented on Dec 3, 2024

    @jeffvestal
    jeffvestal
    on Dec 3, 2024
    Author

    Also, if I wait 5 minutes I get the error again

    {
  "error": {
    "root_cause": [
      {
        "type": "exception",
        "reason": "Amazon Bedrock converse call failed"
      }
    ],
    "type": "exception",
    "reason": "Failed to send request from inference entity id [bedrock_cohere-cmd-r-plus_completion]",
    "caused_by": {
      "type": "exception",
      "reason": "org.elasticsearch.ElasticsearchException: Amazon Bedrock converse call failed",
      "caused_by": {
        "type": "exception",
        "reason": "Amazon Bedrock converse call failed",
        "caused_by": {
          "type": "sdk_client_exception",
          "reason": "Unable to execute HTTP request: event executor terminated",
          "caused_by": {
            "type": "rejected_execution_exception",
            "reason": "event executor terminated"
          }
        }
      }
    }
  },
  "status": 500
}
  6. jonathan-buttner
    added
    :mlMachine learning
    Team:MLMeta label for the ML team
    Feature:GenAIFeatures around GenAI
    and removed
    Team:SearchOrgMeta label for the Search Org (Enterprise Search)
    :SearchOrg/InferenceLabel for the Search Inference team
    Team:Search - Inference
    on Dec 3, 2024

  7. elasticsearchmachine commented on Dec 3, 2024

    @elasticsearchmachine
    elasticsearchmachine
    on Dec 3, 2024
    Collaborator

    Pinging @elastic/ml-core (Team:ML)

  8. ymao1
    self-assigned this
    on Dec 6, 2024
  9. ymao1
    closed this as completedin #118177on Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

:mlMachine learning>bugFeature:GenAIFeatures around GenAITeam:MLMeta label for the ML team

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions