Skip to content

[Meta] support xpack delete and recreate pipeline with same pipeline ID and config string #12387

Description

@kaisecheng

When users delete a pipeline and recreate the same pipeline with the same pipeline ID and config string, Logstash is unable to pick up again because the hash value is the same and the hash value does not consider metadata.

{
    "pipeline": "input { generator { count => 100000 }} output { stdout { codec => \"rubydebug\"}} ",
    "last_modified": "2020-10-01T15:42:30.229Z",
    "pipeline_metadata": { "version": "1"},
    "username": "log.stash",
}

If users delete the above pipeline and recreate it with the same config string with a different metadata {"version": "2"}, Logstash won't recreate the pipeline because the hash function only considers "input { generator { count => 100 }} output { stdout { codec => \"rubydebug\"}} " which is the same as before.
The proposal is to change the hash function to take the metadata into account. Logstash will be able to recompile the pipeline.

Version: 8.0.0-SNAPSHOT
Operating System: Mac
Config File (if you have sensitive info, please remove it): xpack.management.pipeline.id: ["pipeline_id"]
Steps to Reproduce:

  1. enable xpack

  2. config xpack.management.pipeline.id with a valid pipeline id

  3. create a pipeline in elasticsearch version 7.10+
    curl -X PUT -v "localhost:9200/_logstash/pipeline/heartbeat" -H 'Content-Type: application/json' -d '{
    "pipeline": "input { generator { count => 100000 }} output { stdout { codec => "rubydebug"}} ",
    "last_modified": "2020-10-01T15:42:30.229Z",
    "pipeline_metadata": { "version": "1", "type": "logstash_pipeline"},
    "username": "log.stash",
    "pipeline_settings": {"pipeline.batch.delay": "50"}
    }'

  4. delete the same pipeline in elasticsearch
    curl -X DELETE "localhost:9200/_logstash/pipeline/heartbeat"

  5. recreate the pipeline with the same content but "version" update to "2"
    curl -X PUT -v "localhost:9200/_logstash/pipeline/heartbeat" -H 'Content-Type: application/json' -d '{
    "pipeline": "input { generator { count => 100000 }} output { stdout { codec => "rubydebug"}} ",
    "last_modified": "2020-10-01T15:42:30.229Z",
    "pipeline_metadata": { "version": "2", "type": "logstash_pipeline"},
    "username": "log.stash",
    "pipeline_settings": {"pipeline.batch.delay": "50"}
    }'

  6. Logstash unable to output count

Action item:

Activity

  1. changed the title [-]support xpack delete and recreate pipeline with same pipeline ID and config string[/-] [+][Meta] support xpack delete and recreate pipeline with same pipeline ID and config string[/+] on Oct 28, 2020
  2. added 3 commits that reference this issue on Nov 9, 2020
    e53ea5c
    983ba19
    aba562e
  3. added a commit that references this issue on Nov 11, 2020
    96074d6
  4. added a commit that references this issue on Nov 11, 2020
    da7f5f5
  5. added a commit that references this issue on Mar 6, 2025
    606cfe5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions