[doc] Logstash Kubernetes - Stack Monitoring docs by kaisecheng · Pull Request #14696 · elastic/logstash

kaisecheng · 2022-10-25T14:45:48Z

introduction
use metricbeat with autodiscover to monitor multiple Logstashes.
send metrics to external ES cluster
send metrics to Elastic Cloud

For agent/metricbeat monitoring logstash, we have a recipe in beats for stack monitoring while agent is missing this part. Also, we have "Elasticsearch Metrics" integrations for beats but not agent. Agent can monitor Logstash with limitation. If users want to monitor multiple logstashes, they need to assign different service name for each logstash which is not practical. So, in this doc, agent is not mentioned as a solution.

I wanna have stack monitoring in multiple pages as the following structure without success. They are on the same page now. If you know how to split it, please let me know.

Administering
|_ Stack Monitoring
     |_ Ship metrics to external cluster
     |_ Ship metric to Elastic Cloud

Fixes: #14572

robbavey · 2022-10-28T14:36:17Z

jenkins test this please

robbavey

Nice work - I have a couple of questions on structure and formatting

robbavey · 2022-10-28T19:35:16Z

+To enable {logstash-ref}/monitoring-with-metricbeat.html[Stack Monitoring] for {ls}, you need Metricbeat to collect {ls} metrics, {es} to store the metrics and Kibana to view the result.

-Follow these steps to configure monitoring:
+Assuming you have installed ECK, the example modifies the link:https://github.com/elastic/cloud-on-k8s/blob/main/config/recipes/beats/stack_monitoring.yaml[recipe] of Beats stack monitoring. The recipe has initiated a production {es} cluster, a monitoring {es} cluster, {filebeat}, {metricbeat}, a production Kibana and a monitoring Kibana. It monitors {es} and Kibana and send metrics to monitoring cluster.


I wonder if we should call out ECK explicitly, but have an intro section, explaining what we are doing here with these examples:

Something like,

For these examples, we will be modifying the Beats stack monitoring link:https://github.com/elastic/cloud-on-k8s/blob/main/config/recipes/beats/stack_monitoring.yaml[recipe] from the ECK examples, which initiate ...

And then have

=== Stack Monitoring with ECK

Or something along those lines.

Thoughts @karenzone ?

we can add a note of prerequisites of installing ECK which links to the Quick start set up environment

robbavey · 2022-10-28T19:35:53Z

+An important step to making your environment production ready is to configure stack monitoring. Monitoring metrics can be sent to an external resource, such as {ess} or {eck}, so that in the event that any components of your environment become unresponsive, your monitoring data is available.

-An important step to making your environment production ready is to configure stack monitoring. Monitoring logs and metrics data can be sent to an external resource, such as {ess} or {eck}, so that in the event that any components of your environment become unresponsive, your monitoring data is available.
+To enable {logstash-ref}/monitoring-with-metricbeat.html[Stack Monitoring] for {ls}, you need Metricbeat to collect {ls} metrics, {es} to store the metrics and Kibana to view the result.


I wonder if we should start with some prerequisites - for the elastic cloud or external cluster options, we require that the elastic CRD's are installed ahead of time, to ensure that the Beat CRD is available. Maybe call it out ahead of time before we have sections for ECK/Elastic Cloud/External Elasticsearch?

agree, will do it

robbavey · 2022-10-28T20:17:26Z

+
+<2> {metricbeat} scans for the pods with label `app: ls` to collect {ls} metrics.
+
+<3> {metricbeat} logstash module calls metric endpoint from port `9600` for every `10` seconds.


Does it call each logstash every 10 seconds? Or round robin between them, so for 3 logstashes, every 30 seconds?

it calls each logstash every 10 seconds

robbavey · 2022-10-28T20:18:14Z

+...
+--
+
+Provide the `cluster_uuid` of the production {es} cluster to `monitoring.cluster_uuid` in logstash.yml.


Do we have a link anywhere on how to do that?

I added instructions of getting uuid

Co-authored-by: Rob Bavey <rob.bavey@elastic.co>

github-actions · 2022-11-01T11:11:08Z

📃 DOCS PREVIEW ✨ https://logstash_14696.docs-preview.app.elstc.co/diff

add instruction of getting cluster_uuid

github-actions · 2022-11-01T13:17:03Z

📃 DOCS PREVIEW ✨ https://logstash_14696.docs-preview.app.elstc.co/diff

github-actions · 2022-11-01T13:55:37Z

📃 DOCS PREVIEW ✨ https://logstash_14696.docs-preview.app.elstc.co/diff

robbavey · 2022-11-03T13:29:40Z

+[[ls-k8s-monitor-external]]
+==== Ship metrics to external {es} cluster
+
+NOTE: The prerequisite of the example is having ECK installed. Checkout <<qs-set-up>>


Is it a prerequisite to have ECK installed? Or just the CRDs?

The example involves the deployment of stack, so it is ECK

I think I get confused between Metrics can be sent to an {es} cluster that is not managed by ECK, and the prerequisite of ECK for this - one use case is for a new logstash cluster sending to a pre-existing monitoring cluster, not migrated to ECK

You are right, to config the external ES we just need CRDs. I mess up with another example.

robbavey · 2022-11-03T13:31:12Z

+[[ls-k8s-monitor-elastic-cloud]]
+==== Ship metrics to Elastic Cloud
+
+NOTE: The prerequisite of the example is having ECK installed. Checkout <<qs-set-up>>


Is it a prerequisite to have ECK installed? Or just the CRDs?

I am not sure how much the operator has been involved in associating Beat with Elasticsearch. I am hesitant to suggest users install CRD only. So, the prerequisite is ECK.

Do you mean from the perspective of whether the Beats CRD can be used in isolation, without any of the other components of the stack?

It feels like it might be a common use case to not want to install any of the other stack elements, and just use K8s as a container for Logstash (and metricbeat to monitor it), and to send to Elastic Cloud. In which case we'd need only the sections with metadata.name: metricbeat sections, and not any of the filebeat, elasticsearch or kibana sections. And we wouldn't need the elasticsearch and kibana modules in the metricbeat definition.
And it might make sense to add this as a recipe to refer back to, to reduce the confusion with the rest of the file?

WDYT?

I probably mess up the example when I replied. This should be valid to say CRDs as a prerequisite.

My understanding of ECK includes two parts. 1. CRD 2. operator. In order to have CRDs work with full functionality, we need both. The part I was uncertain is which features/configs need operator to work, so hesitate to just suggest installing CRD. But in this feature connecting to Elastic Cloud, I think we are safe to require CRD only.

Co-authored-by: Rob Bavey <rob.bavey@elastic.co>

github-actions · 2022-11-03T13:52:36Z

📃 DOCS PREVIEW ✨ https://logstash_14696.docs-preview.app.elstc.co/diff

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

github-actions · 2022-11-04T13:41:07Z

📃 DOCS PREVIEW ✨ https://logstash_14696.docs-preview.app.elstc.co/diff

…oc_k8s_stack_monitoring # Conflicts: # docsk8s/administering/ls-k8s-stack-monitoring-cloud.asciidoc

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

…oc_k8s_stack_monitoring

github-actions · 2022-11-04T15:41:48Z

📃 DOCS PREVIEW ✨ https://logstash_14696.docs-preview.app.elstc.co/diff

robbavey

Really close!

robbavey · 2022-11-04T17:29:40Z

+[[ls-k8s-monitor-config-ls]]
+===== Configure {ls}
+
+Add label `app: ls` to `Deployment` for autodiscover.


Do we need to mention StatefulSet here too?

I think it is optional

Co-authored-by: Rob Bavey <rob.bavey@elastic.co>

github-actions · 2022-11-04T19:00:53Z

📃 DOCS PREVIEW ✨ https://logstash_14696.docs-preview.app.elstc.co/diff

robbavey

LGTM

Nice job!

karenzone

I revamped a suggestion from the previous review based on the clarification you provided (that "it" means "Metricbeat." Other than adding/committing that one, LGTM! 🚀

karenzone · 2022-11-04T19:53:43Z

+
+NOTE: The prerequisite of the example is having CRDs installed. Checkout <<qs-set-up>>
+
+Metrics can be sent to an {es} cluster that is not managed by ECK. To configure it, remove the `elasticsearchRef` from the specification and include an output configuration in the `spec.config`.


Thanks for the info. I updated the suggestion to clarify. :-)

kaisecheng added 4 commits October 17, 2022 16:22

[DOC] k8s stack monitoring

358865f

use metricbeat autodiscover to monitor multiple logstash

1f4c9dd

update stack monitoring

d7049e4

wording

f2779a4

roaksoax added the status:work-in-progress label Oct 25, 2022

roaksoax assigned kaisecheng Oct 25, 2022

kaisecheng added 2 commits October 26, 2022 15:42

add number indicator

570aeaa

split stack monitoring to pages

a1ee536

kaisecheng marked this pull request as ready for review October 26, 2022 15:20

kaisecheng requested a review from robbavey October 26, 2022 15:29

roaksoax added status:needs-review and removed status:work-in-progress labels Oct 26, 2022

set kibana port forwarding

e33f1eb

roaksoax assigned karenzone Oct 27, 2022

robbavey reviewed Oct 28, 2022

View reviewed changes

roaksoax changed the title ~~[Doc] k8s stack monitoring~~ [doc] Logstash Kubernetes - Stack Monitoring docs Oct 31, 2022

kaisecheng and others added 2 commits November 1, 2022 10:58

Update docsk8s/administering/ls-k8s-stack-monitoring-cloud.asciidoc

38d3bdc

Co-authored-by: Rob Bavey <rob.bavey@elastic.co>

Update docsk8s/administering/ls-k8s-stack-monitoring.asciidoc

d492209

Co-authored-by: Rob Bavey <rob.bavey@elastic.co>

update wording

ecbe8a9

add instruction of getting cluster_uuid

add prerequisite note

8019694

kaisecheng requested a review from robbavey November 1, 2022 13:56

robbavey requested changes Nov 3, 2022

View reviewed changes

kaisecheng and others added 3 commits November 3, 2022 13:40

Update docsk8s/administering/ls-k8s-stack-monitoring-external.asciidoc

4c49b46

Co-authored-by: Rob Bavey <rob.bavey@elastic.co>

Update docsk8s/administering/ls-k8s-stack-monitoring.asciidoc

46ae424

Co-authored-by: Rob Bavey <rob.bavey@elastic.co>

Update docsk8s/administering/ls-k8s-stack-monitoring.asciidoc

4247d3c

Co-authored-by: Rob Bavey <rob.bavey@elastic.co>

kaisecheng and others added 6 commits November 4, 2022 13:26

Update docsk8s/administering/ls-k8s-stack-monitoring.asciidoc

1540869

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

Update docsk8s/administering/ls-k8s-stack-monitoring-cloud.asciidoc

72ebd74

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

Update docsk8s/administering/ls-k8s-stack-monitoring.asciidoc

622c058

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

Update docsk8s/administering/ls-k8s-stack-monitoring.asciidoc

9b868b9

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

Update docsk8s/administering/ls-k8s-stack-monitoring.asciidoc

f17303a

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

Update docsk8s/administering/ls-k8s-stack-monitoring.asciidoc

1e340ac

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

kaisecheng and others added 9 commits November 4, 2022 15:16

Change prerequisite from Note to Tip

b83e6a9

Merge remote-tracking branch 'origin/doc_k8s_stack_monitoring' into d…

c7c5a77

…oc_k8s_stack_monitoring # Conflicts: # docsk8s/administering/ls-k8s-stack-monitoring-cloud.asciidoc

Update docsk8s/administering/ls-k8s-stack-monitoring-cloud.asciidoc

df3b935

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

Update docsk8s/administering/ls-k8s-stack-monitoring.asciidoc

ef769ba

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

Update docsk8s/administering/ls-k8s-stack-monitoring.asciidoc

6c53616

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

Update docsk8s/administering/ls-k8s-stack-monitoring.asciidoc

8cbac85

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

Update docsk8s/administering/ls-k8s-stack-monitoring.asciidoc

8159180

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

remove it

4a7def0

Merge remote-tracking branch 'origin/doc_k8s_stack_monitoring' into d…

efd999d

…oc_k8s_stack_monitoring

kaisecheng requested a review from karenzone November 4, 2022 15:32

kaisecheng requested a review from robbavey November 4, 2022 16:46

robbavey reviewed Nov 4, 2022

View reviewed changes

kaisecheng and others added 2 commits November 4, 2022 18:48

Update docsk8s/administering/ls-k8s-stack-monitoring-cloud.asciidoc

82e5e54

Co-authored-by: Rob Bavey <rob.bavey@elastic.co>

Update docsk8s/administering/ls-k8s-stack-monitoring-external.asciidoc

e3de63e

Co-authored-by: Rob Bavey <rob.bavey@elastic.co>

robbavey approved these changes Nov 4, 2022

View reviewed changes

karenzone approved these changes Nov 4, 2022

View reviewed changes

roaksoax added status:approved and removed status:changes-requested labels Nov 4, 2022

kaisecheng merged commit 6847ad8 into elastic:main Nov 4, 2022

roaksoax removed the status:approved label Nov 4, 2022

roaksoax added the docs label Dec 13, 2023


		<2> {metricbeat} scans for the pods with label `app: ls` to collect {ls} metrics.

		<3> {metricbeat} logstash module calls metric endpoint from port `9600` for every `10` seconds.


		NOTE: The prerequisite of the example is having CRDs installed. Checkout <<qs-set-up>>

		Metrics can be sent to an {es} cluster that is not managed by ECK. To configure it, remove the `elasticsearchRef` from the specification and include an output configuration in the `spec.config`.

Conversation

kaisecheng commented Oct 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robbavey commented Oct 28, 2022

Uh oh!

robbavey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Nov 1, 2022

Uh oh!

github-actions Bot commented Nov 1, 2022

Uh oh!

github-actions Bot commented Nov 1, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Nov 3, 2022

Uh oh!

github-actions Bot commented Nov 4, 2022

Uh oh!

github-actions Bot commented Nov 4, 2022

Uh oh!

robbavey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Nov 4, 2022

Uh oh!

robbavey left a comment

Choose a reason for hiding this comment

Uh oh!

karenzone left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

kaisecheng commented Oct 25, 2022 •

edited

Loading