Skip to content

[Obs AI Assistant] semantic_text migration may run indefinitely leading to OOM #220339

Description

@sorenlouv

The semantic text migration can cause excessive traffic to a cluster and might eventually cause the node to crash due to OOM.

The root cause is, that it is possible to create knowledge base entries without content, aka an empty text field. This was made possible in #202279.

The migration script does not handle this scenario and will indefinitely update the same document. In some cases we saw ~1 million document updates per day on a single cluster due to 1 document having empty text.
Because the document update involves semantic_text an ML node is kept warm further increasing the cost of this.

Activity

  1. added
    bugFixes for quality problems that affect the customer experience
    on May 7, 2025
  2. self-assigned this
    on May 7, 2025
  3. elasticmachine commented on May 7, 2025

    @elasticmachine
    Contributor

    Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant)

  4. added theissue type on May 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugFixes for quality problems that affect the customer experience

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions