Add prometheus and grafana make commands for local environment by fosterseth · Pull Request #12240 · ansible/awx

fosterseth · 2022-05-16T18:42:57Z

SUMMARY

make prometheus and make grafana will start local docker containers that integrate with a running awx instance.

ISSUE TYPE

Feature Pull Request

COMPONENT NAME

API

AWX VERSION

awx: 21.0.1.dev107+gb03dbc68c2

AlanCoding · 2022-05-31T18:15:49Z

 {% if control_plane_node_count|int > 1 %}
  haproxy:
-    image: haproxy
+    image: haproxy:2.3


looks like bleed over from another open PR you have.

AlanCoding · 2022-05-31T18:19:18Z

+
+1. Change the `username` and `password` in `tools/prometheus/prometheus.yml`. You can also change the scrape interval.
+2. (optional) if you are in cluster environment, you can change the target to `haproxy:8043` so that the incoming prometheus requests go through the load balancer. Leaving it set to `awx1` also works.
+3. run `make prometheus`


like, in a new terminal tab or something?

AlanCoding · 2022-05-31T18:29:42Z

+          "refId": "B"
+        }
+      ],
+      "title": "jobs running pending",


the title confuses me somewhat

john-westcott-iv · 2022-05-31T18:32:59Z

+Prometheus is a metrics collecting tool, and we support prometheus formatted data at the `api/v2/metrics` endpoint.
+
+1. Change the `username` and `password` in `tools/prometheus/prometheus.yml`. You can also change the scrape interval.
+2. (optional) if you are in cluster environment, you can change the target to `haproxy:8043` so that the incoming prometheus requests go through the load balancer. Leaving it set to `awx1` also works.


Are these new tools for dev testing only or are they actually intended to be run against something like a cluster?

If we are leaving this in here you are in a clustered environment.

intended for dev / testing usage

john-westcott-iv · 2022-05-31T18:35:08Z

+1. Change the `username` and `password` in `tools/prometheus/prometheus.yml`. You can also change the scrape interval.
+2. (optional) if you are in cluster environment, you can change the target to `haproxy:8043` so that the incoming prometheus requests go through the load balancer. Leaving it set to `awx1` also works.
+3. run `make prometheus`
+4. navigate to `http://localhost:9090/targets` and check that the metrics endpoint State is Up.


Do you mean the metrics API endpoint or an endpoint under prometheus (this is right after the prometheus URL).

Also, would this be "localhost" or would that depend on your installation? In other places in this document we talk about "container_reference" and how one might get that.

john-westcott-iv · 2022-05-31T18:39:30Z

 prometheus:
-	docker run -u0 --net=tools_default --link=`docker ps | egrep -o "tools_awx(_run)?_([^ ]+)?"`:awxweb --volume `pwd`/tools/prometheus:/prometheus --name prometheus -d -p 0.0.0.0:9090:9090 prom/prometheus --web.enable-lifecycle --config.file=/prometheus/prometheus.yml
+	docker volume create prometheus
+	docker run -d --rm --net=_sources_default --link=awx_1:awx1 --volume prometheus-storage:/prometheus --volume `pwd`/tools/prometheus:/etc/prometheus --name prometheus -p 9090:9090 prom/prometheus
+
+grafana:
+	docker volume create grafana
+	docker run -d --rm --net=_sources_default --volume grafana-storage:/var/lib/grafana --volume `pwd`/tools/grafana:/etc/grafana/provisioning --name grafana -p 3001:3000 grafana/grafana-enterprise


Instead of individual targets should we move these into the model like we did for LDAP/SPLUNK/etc where you would do something like: GRAFANA=True make docker-compose?

I would ultimately prefer this, but I think it makes sense to merge it with the current scheme and work on this later.

The biggest pain point for me of the current approach is that it doesn't show startup logs, and integrating it into docker-compose would fix that.

john-westcott-iv · 2022-05-31T18:45:34Z

-
  - job_name: 'awx'
+    static_configs:
+      - targets: ['awx1:8043']  # or haproxy:8043 in cluster env


Should this be awx1 or tools_awx_1?

prometheus expects a real dns name here, so underscores aren't allowed.

that is why I need to alias awx_1 to awx1 when starting the prometheus container

AlanCoding · 2022-05-31T19:09:40Z

Link candidate fix AlanCoding@12c4807

kdelee

Works for me out of the box now!

This is after deleting all containers/volumes and starting fresh

AlanCoding · 2022-06-01T14:41:46Z

 prometheus:
-	docker run -u0 --net=tools_default --link=`docker ps | egrep -o "tools_awx(_run)?_([^ ]+)?"`:awxweb --volume `pwd`/tools/prometheus:/prometheus --name prometheus -d -p 0.0.0.0:9090:9090 prom/prometheus --web.enable-lifecycle --config.file=/prometheus/prometheus.yml
+	docker volume create prometheus
+	docker run -d --rm --net=_sources_default --link=awx_1:awx1 --volume prometheus-storage:/prometheus --volume `pwd`/tools/prometheus:/etc/prometheus --name prometheus -p 9090:9090 prom/prometheus


reminder for later - we need a target for something like

docker volume rm prometheus prometheus-storage grafana grafana-storage

fosterseth force-pushed the make_prometheus_grafana branch from 0b171a3 to ae805e2 Compare May 16, 2022 18:51

awxbot added type:enhancement component:api labels May 16, 2022

github-actions Bot removed the component:api label May 16, 2022

fosterseth force-pushed the make_prometheus_grafana branch 2 times, most recently from 1f3b6ca to 568f93c Compare May 16, 2022 21:38

AlanCoding requested a review from john-westcott-iv May 31, 2022 18:15

AlanCoding reviewed May 31, 2022

View reviewed changes

john-westcott-iv reviewed May 31, 2022

View reviewed changes

Add make prometheus and make grafana commands to dev environment

6f68f3c

fosterseth force-pushed the make_prometheus_grafana branch from 8579cf8 to 6f68f3c Compare May 31, 2022 21:07

fosterseth added 3 commits May 31, 2022 17:12

haproxy

7fe98a6

maxconn

5831949

typo

9364c8e

kdelee approved these changes May 31, 2022

View reviewed changes

AlanCoding approved these changes Jun 1, 2022

View reviewed changes

AlanCoding reviewed Jun 1, 2022

View reviewed changes

AlanCoding mentioned this pull request Jun 1, 2022

apply start task limit to queries of pending jobs #12214

Closed

fosterseth merged commit b548ad2 into ansible:devel Jun 1, 2022

fosterseth deleted the make_prometheus_grafana branch March 11, 2025 17:42

Conversation

fosterseth commented May 16, 2022

SUMMARY

ISSUE TYPE

COMPONENT NAME

AWX VERSION

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlanCoding commented May 31, 2022

Uh oh!

kdelee left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants