Feature: flow metrics integration - #14518

Release notes

Adds pipeline "flow" metrics to the node_stats API for each pipeline, which includes the current and lifetime rates for five key pipeline metrics: input_throughput, filter_throughput, output_throughput, queue_backpressure, and worker_concurrency.

What does this PR do?

Implements Phase 0 and 1.A of #14463, providing each pipeline's current and lifetime rates for 5 key metrics:

• input_throughput: pipelines.*.events.in / second of pipeline uptime
• filter_throughput: pipelines.*.events.filtered / second of pipeline uptime
• output_throughput: pipelines.*.events.out / second of pipeline uptime
• queue_backpressure: pipelines.*.queue_push_duration / second of pipeline uptime
• worker_concurrency: duration / second of pipeline uptime

This Pull-Request is a combination of the efforts reviewed at a high-level in #14509 and #14514 for inclusion into this feature branch, and is a place for us to further review and document the now-integrated components.

Why is it important/What is the impact to the user?

As described in #14463, notably:

Problem Statement

It is often difficult to understand the health of a pipeline, including whether it is exerting or propagating back-pressure or otherwise staying reasonably “caught up” with its inputs. The cumulative-value metrics exposed by the API are not typically useful on their own, and when we are debugging a pipeline we often require a pair of captures and manual math in order to see the flow of events through the pipeline.

We want to provide better visibility into the flow of events through a Logstash pipeline in order to easier pinpoint the source of degraded status.

Scope and Goals

We will provide visibility into the current, recent, and lifetime flow rates of pipeline-level metrics in the /_node/stats API, so that our users can have more contextual information when analyzing the throughput of a pipeline.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files (and/or docker env variables)
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• periodic poller warns if it is unable to keep up with configured polling frequency
• expose our pipeline's new uptime_in_millis to the API response.

How to test this PR locally

1. Run a pipeline that handles jittery input data.

   ruby -e '100.times.map { |j| Thread.new(j) { |k| 100000.times { |i| $stdout.write("ok[#{k}/#{i}]\n"); (i%17).zero? && sleep(Random.rand(10)) } } }.map(&:join)' | bin/logstash -e 'input { stdin {} } output { sink {} }'
   

2. once the pipeline is started, query the node stats API, and observe the new pipelines.main.flow metrics, and how they relate to the cumulative-value numbers they are tracking.