The semantic text migration can cause excessive traffic to a cluster and might eventually cause the node to crash due to OOM.
The root cause is, that it is possible to create knowledge base entries without content, aka an empty text field. This was made possible in #202279.
The migration script does not handle this scenario and will indefinitely update the same document. In some cases we saw ~1 million document updates per day on a single cluster due to 1 document having empty text.
Because the document update involves semantic_text an ML node is kept warm further increasing the cost of this.
The semantic text migration can cause excessive traffic to a cluster and might eventually cause the node to crash due to OOM.
The root cause is, that it is possible to create knowledge base entries without content, aka an empty
textfield. This was made possible in #202279.The migration script does not handle this scenario and will indefinitely update the same document. In some cases we saw ~1 million document updates per day on a single cluster due to 1 document having empty
text.Because the document update involves
semantic_textan ML node is kept warm further increasing the cost of this.