Skip to content

Prometheus server, exporter and altermanager roles - #1310

Open
pedroluislopez wants to merge 33 commits into
debops:masterfrom
Neuromobile:prometheus
Open

pedroluislopez wants to merge 33 commits into
debops:masterfrom
Neuromobile:prometheus

Conversation

@pedroluislopez

@pedroluislopez pedroluislopez commented Mar 30, 2020

Copy link
Copy Markdown
Member
  • Prometheus server is configured with file discovery by default.
  • Prometheus exporter can be configure and install exporter debian packages, exporter releases from repo projects and can configure built-in service exporters (like rabbitmq server 3.8).
  • Include prometheus alertmanager role.
  • Prometheus exporter and alertmanager configure file discovery files in server.
  • Allow customize exporters startup variables.
  • PKI: Support TLS and cert authentication with Ghostunnel https://github.com/square/ghostunnel. between prometheus server and exporters and alertmanagers.
  • Configured Nginx basic auth for Prometheus server front and alertmanager front.
  • Support for prometheus rules.

Missing:

  • Rules files.
  • Prometheus alertmanager role.
  • HTTP reverse proxy for SSL and auth. In prometheus recommend https://github.com/square/ghostunnel for production use. Maybe it's a better option than installing nginx.
  • Write documentation.

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
…access to sockets

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
…ault configuratio files.

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
@reixd

reixd commented Apr 14, 2020

Copy link
Copy Markdown
Contributor

Any news on this. I really would like to have Prom working as a role

@ganto

ganto commented Apr 15, 2020

Copy link
Copy Markdown
Contributor

I was quickly trying to compare this PR with #160 which also tries to add Prometheus support to DebOps:

In your code, I very like the facts-based integration of the exporters with the Prometheus server.

I'm currently running a setup based on the code in #160 and use a few features that are missing here. Especially these are:

  • Alertmanager integration
  • Nginx reverse proxy for SSL and basic auth in front of exporters and Prometheus server

These features are imho essential (at least for my setup).

What I'm still missing in both PR is the customization of startup arguments for the exporters and general customization of exporters. E.g. I'm running a systemd timer to provide data via node exporter collector.textfile.directory.

The Ansible code looks decent. However, I personally don't like the include_role tasks. I don't know if this changed recently but in the past DebOps wouldn't do this and use dependencies in the meta file instead.

@pedroluislopez pedroluislopez changed the title Prometheus server and prometheus exporter roles [WIP] Prometheus server and prometheus exporter roles Apr 15, 2020
@pedroluislopez

pedroluislopez commented Apr 15, 2020

Copy link
Copy Markdown
Member Author

I've set the PR in WIP.

So, missing features:

  • Rules files.
  • Prometheus alertmanager role.
  • HTTP reverse proxy for SSL and auth. In prometheus recommend https://github.com/square/ghostunnel for production use. Maybe it's a better option than installing nginx, @ganto what do you think?
  • Write documentation.

@ganto With this roles you can customize of startup arguments for the exporters. For example:

prometheus_exporter__args:

  - name: 'node'
    options:
      - collector.textfile.directory: '/var/lib/prometheus/node-exporter'

The recent DebOps code use include_role for secret and ansible_plugins role import in other roles.

@pedroluislopez

pedroluislopez commented Apr 15, 2020

Copy link
Copy Markdown
Member Author

Inventory examples.

Prometheus server:

prometheus_server__scrape_configs_configuration:

  - job_name: 'nginx'
    scheme: '{{ "https" if prometheus_server__pki|bool else "http" }}'
    tls_config: '{{ prometheus_server__tls_config if prometheus_server__pki|bool else {} }}'
    file_sd_configs:
      - files:
        - '/etc/prometheus/file_sd_configs.d/nginx.*.json'
    relabel_configs:
      - source_labels: ['__address__']
        separator:     ':'
        regex:         '([^.:]*)[.].*:(.*)' # Get hostname from address
        target_label:  'instance'
        replacement:   '${1}:${2}'

  - job_name: 'phpfpm'
    scheme: '{{ "https" if prometheus_server__pki|bool else "http" }}'
    tls_config: '{{ prometheus_server__tls_config if prometheus_server__pki|bool else {} }}'
    file_sd_configs:
      - files:
        - '/etc/prometheus/file_sd_configs.d/phpfpm.*.json'
    relabel_configs:
      - source_labels: ['__address__']
        separator:     ':'
        regex:         '([^.:]*)[.].*:(.*)' # Get hostname from address
        target_label:  'instance'
        replacement:   '${1}:${2}'

  - job_name: 'mysqld'
    scheme: '{{ "https" if prometheus_server__pki|bool else "http" }}'
    tls_config: '{{ prometheus_server__tls_config if prometheus_server__pki|bool else {} }}'
    file_sd_configs:
      - files:
        - '/etc/prometheus/file_sd_configs.d/mysqld.*.json'
    relabel_configs:
      - source_labels: ['__address__']
        separator:     ':'
        regex:         '([^.:]*)[.].*:(.*)' # Get hostname from address
        target_label:  'instance'
        replacement:   '${1}:${2}'

  - job_name: 'redis'
    scheme: '{{ "https" if prometheus_server__pki|bool else "http" }}'
    tls_config: '{{ prometheus_server__tls_config if prometheus_server__pki|bool else {} }}'
    file_sd_configs:
      - files:
        - '/etc/prometheus/file_sd_configs.d/redis.*.json'
    relabel_configs:
      - source_labels: ['__address__']
        separator:     ':'
        regex:         '([^.:]*)[.].*:(.*)' # Get hostname from address
        target_label:  'instance'
        replacement:   '${1}:${2}'

  - job_name: 'mongodb'
    scheme: '{{ "https" if prometheus_server__pki|bool else "http" }}'
    tls_config: '{{ prometheus_server__tls_config if prometheus_server__pki|bool else {} }}'
    file_sd_configs:
      - files:
        - '/etc/prometheus/file_sd_configs.d/mongodb.*.json'
    relabel_configs:
      - source_labels: ['__address__']
        separator:     ':'
        regex:         '([^.:]*)[.].*:(.*)' # Get hostname from address
        target_label:  'instance'
        replacement:   '${1}:${2}'

  - job_name: 'rabbitmq'
    scheme: '{{ "https" if prometheus_server__pki|bool else "http" }}'
    tls_config: '{{ prometheus_server__tls_config if prometheus_server__pki|bool else {} }}'
    file_sd_configs:
      - files:
        - '/etc/prometheus/file_sd_configs.d/rabbitmq.*.json'
    relabel_configs:
      - source_labels: ['__address__']
        separator:     ':'
        regex:         '([^.:]*)[.].*:(.*)' # Get hostname from address
        target_label:  'instance'
        replacement:   '${1}:${2}'

All nodes (by default it installs and configure node exporter):

prometheus_exporter__server: 'prometheus.domain.com
prometheus_exporter__allow: [ '<IP prometheus server>' ]

Nginx and PHP-FPM:

prometheus_exporter__append_groups: [ 'www-data' ]

prometheus_exporter__group_exporters: [ 'nginx' ]

prometheus_exporter__group_release_exporters:

  - name: 'phpfpm'
    resource: 'https://github.com/Lusitaniae/phpfpm_exporter/releases/download/v0.5.0/phpfpm_exporter-0.5.0.linux-amd64.tar.gz'
    archive: True
    binary: 'phpfpm_exporter-0.5.0.linux-amd64/phpfpm_exporter'

prometheus_exporter__group_ports_map:

  phpfpm: '9253'

prometheus_exporter__args:

  - name: 'nginx'
    options:
      - nginx.scrape-uri: 'https://{{ ca_migrations_fqdn }}/nginx_status'

  - name: 'phpfpm'
    options:
      - phpfpm.socket-paths:
           - '/run/php5.6-fpm-migrations.sock'
      - phpfpm.status-path: '/status.php'

MariaDB

prometheus_exporter__group_exporters: [ 'mysqld' ]

Redis

prometheus_exporter__group_release_exporters:

  - name: 'redis'
    resource: 'https://github.com/oliver006/redis_exporter/releases/download/v1.5.3/redis_exporter-v1.5.3.linux-amd64.tar.gz'
    archive: True
    binary: 'redis_exporter-v1.5.3.linux-amd64/redis_exporter'

prometheus_exporter__group_ports_map:

  redis: '9121'

prometheus_exporter__args:

  - name: 'redis'
    options:
      - redis.password: '{{ lookup("password", secret + "/redis/clusters/" + ansible_domain + "/password") }}'

MongoDB

prometheus_exporter__group_exporters: [ 'mongodb' ]

prometheus_exporter__args:

  - name: 'mongodb'
    options:
      - mongodb.uri: '{{ """mongodb://monitor:" +
                         lookup("password", secret + "/mongodb/" + ansible_fqdn + "/credentials/monitor/password") +
                         "@localhost:27017""" }}'

RabbitMQ 3.8

prometheus_exporter__pre_installed_exporters: [ 'rabbitmq' ]

prometheus_exporter__group_ports_map:

  rabbitmq: '15672'

RabbitMQ < 3.8

prometheus_exporter__group_release_exporters:

  - name: 'rabbitmq'
    resource: 'https://github.com/kbudde/rabbitmq_exporter/releases/download/v1.0.0-RC6.1/rabbitmq_exporter-1.0.0-RC6.1.linux-amd64.tar.gz'
    archive: True
    binary: 'rabbitmq_exporter-1.0.0-RC6.1.linux-amd64/rabbitmq_exporter'

prometheus_exporter__group_ports_map:

  rabbitmq: '9419'

prometheus_exporter__env:

  - name: 'rabbitmq'
    options:
      - RABBIT_USER: 'sensu'
        RABBIT_PASSWORD: '{{ lookup("password", secret + "/rabbitmq_server/accounts/sensu/password") }}'
        PUBLISH_ADDR: '{{ "localhost" if prometheus_exporter__pki|bool else prometheus_exporter__bind }}'
        PUBLISH_PORT: '{{ "3419" if prometheus_exporter__pki|bool else "9419" }}'

EDIT: [prometheus_exporter] Install repo exporter releases instead of compile go
EDIT 2: [prometheus_server] [prometheus_exporter] Add SSL feature with authentication support

@ganto

ganto commented Apr 16, 2020

Copy link
Copy Markdown
Contributor

@ganto With this roles you can customize of startup arguments for the exporters. For example:

Neat, I missed that 👍

The recent DebOps code use include_role for secret and ansible_plugins role import in other roles.

Hmn, do you have an example? Can't find a single include_role statement in the current master branch.

@pedroluislopez

Copy link
Copy Markdown
Member Author

@ganto With this roles you can customize of startup arguments for the exporters. For example:

Neat, I missed that +1

The recent DebOps code use include_role for secret and ansible_plugins role import in other roles.

Hmn, do you have an example? Can't find a single include_role statement in the current master branch.

073419e

…le go

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
…tication support

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
…rver, localhost dosen't work with TLS

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
@pedroluislopez

Copy link
Copy Markdown
Member Author

Prometheus rules example:

prometheus_server__rules:

  - name: 'example'
    rules:
      - alert: PrometheusTargetMissing
        expr: up == 0
        for: 5m
        labels:
          severity: error
        annotations:
          summary: !unsafe "Prometheus target missing (instance {{ $labels.instance }})"
          description: !unsafe "A Prometheus target has disappeared. An exporter might be crashed.\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
@pedroluislopez

Copy link
Copy Markdown
Member Author

Prometheus alertmanager inventory example:

prometheus_alertmanager__server: 'prometheus.domain.com'
prometheus_alertmanager__allow: [ '<IP prometheus server>' ]
prometheus_alertmanager__global_configuration:
  smtp_from: prometheus@domain.com'
  smtp_smarthost: '...'
  smtp_auth_username: '...'
  smtp_auth_password: '...'
prometheus_alertmanager__receivers_configuration:
  - name: 'default-receiver'
    email_configs:
      - to: 'op@domain.com'

@pedroluislopez

Copy link
Copy Markdown
Member Author

Any news on this. I really would like to have Prom working as a role

Now the roles are completed. Next days i will write documentation.

…ocumentation

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
@pedroluislopez

Copy link
Copy Markdown
Member Author

Refactoring: use golang role for install exporters.
New examples in documentation.

ypid added 6 commits June 14, 2020 17:29
Mechanical edit done by (debops-optimize):

```Shell
git ls-files -z "ansible/roles/prometheus_*" | xargs --null -I '{}' find './{}' -type f -print0 | \
    xargs --null perl -i -p0e '
        s~
            ## Use Ansible 2.8 default shortcut: True or False

            ## Line: .*True
            \(?True\n?

            ## Line: if (ansible_local|d() and ansible_local.pki|d() and
            \s+if\s*\((?<var_ref1>ansible_local)\s*\|\s*d(?:efault)?\(\)\s+and\s+(?<var_ref2>\k<var_ref1>\.\w+)\s*\|\s*d(?:efault)?\(\)\s+and\n?
            ## var_ref1: ansible_local
            ## var_ref2: ansible_local.pki

            ## Line: ansible_local.pki.enabled|d()|bool)
            \s+(?<expression>\k<var_ref2>\.\w+\s*\|\s*d(?:efault)?\(\))\)\n?
            ## expression: ansible_local.pki.enabled|d()|bool

            ## Line: else False
            \s+else\s+False\)?(?:\s*\|\s*bool)?
         ~$+{expression}~xg;

        s~
            ## Use Ansible 2.8 default shortcut: Thing if defined else fallback

            ## Line: .*ansible_local.core.domain
            \(?(?<var_ref3>(?<var_ref2>(?<var_ref1>ansible_local)\.\w+)\.\w+)\n?
            ## var_ref1: ansible_local
            ## var_ref2: ansible_local.core
            ## var_ref3: ansible_local.core.domain

            ## Line: if (ansible_local|d() and ansible_local.core|d() and
            \s+if\s+\(\k<var_ref1>\s*\|\s*d(?:efault)?\(\)\s+and\s+\k<var_ref2>\s*\|\s*d(?:efault)?\(\)\s+and\n?

            ## Line: ansible_local.core.domain|d())
            \s+\k<var_ref3>(?:\s*\|\s*d(?:efault)?\(\))?\)\n?

            ## Line: else ansible_domain
            \s+else\s+\(?(?<else_exp>[^\n]+?)\)?\s+(?<end_marker>}}|%})
            ## else_exp: ansible_domain
        ~$+{var_ref3}|d($+{else_exp}) $+{end_marker}~xg;

        s~
            ## Use Ansible 2.8 default shortcut: More generic catch-all as last resort.

            ## Line: ansible_local|d() and ansible_local.pki|d() and
            (?<var_ref1>ansible_local)\s*\|\s*d(?:efault)?\(\)\s+and\s+(?<var_ref2>\k<var_ref1>\.\w+)\s*\|\s*d(?:efault)?\(\)\s+and\n?
            ## var_ref1: ansible_local
            ## var_ref2: ansible_local.pki

            ## Line: ansible_local.pki.enabled|d()|bool
            \s+(?<expression>\k<var_ref2>\.\w+\s*\|\s*d(?:efault)?\(\))
            ## expression: ansible_local.pki.enabled|d()
         ~$+{expression}~xg;
    '
```
@ypid

ypid commented Jun 14, 2020

Copy link
Copy Markdown
Member

LGTM, thanks. I made a few improvements in Neuromobile#1. Note that I only tested the prometheus_server role (currently all I need).

pedroluislopez and others added 2 commits June 15, 2020 07:30
A few improvements for the Prometheus roles.
…depend server

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
Comment thread docs/ansible/roles/prometheus_server/getting-started.rst Outdated
Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
@julien-lecomte

Copy link
Copy Markdown
Contributor

Prometheus provides many different exporters other than the 'node' one (blackbox-exporter, etc...)
Shouldn't prometheus_exporter be better name with the full name prometheus_node_exporter ?

@pedroluislopez

pedroluislopez commented Sep 7, 2020

Copy link
Copy Markdown
Member Author

Prometheus provides many different exporters other than the 'node' one (blackbox-exporter, etc...)
Shouldn't prometheus_exporter be better name with the full name prometheus_node_exporter ?

The prometheus_exporter role allow install different exporters, and install node exporter by default. You can see a lot of examples in documentation.

@julien-lecomte

Copy link
Copy Markdown
Contributor

Any news on this?

author: 'Pedro Luis López'
description: 'Install and manage Prometheus alertmanager'
company: 'DebOps'
license: 'GPL-3.0-only'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But marked "GPL-3.0-or-later" in the header.

@pedroluislopez

Copy link
Copy Markdown
Member Author

Any news on this?

I have to do refactoring. @drybjed, we need first finish/approve Ghostunnel PR #1600, then I have to do refactoring in this PR to use the Ghostunnel role.

Signed-off-by: Pedro Luis López Sánchez <plopez@neuromobilemarketing.com>
@hungpr0

hungpr0 commented Jul 27, 2021

Copy link
Copy Markdown
Contributor

Thanks a lot for the role.
Just want to ask if you can add task name on 2 import_role. ansible-lint is complaining. They should be:

- name: Import custom Ansible plugins
  import_role:
    name: 'ansible_plugins'

- name: Import DebOps secret role
  import_role:
    name: 'secret'

@hungpr0

hungpr0 commented Aug 5, 2021

Copy link
Copy Markdown
Contributor

@pedroluislopez
It seems prometheus_exporter doesn't install any Debian packages, right? I don't see any apt modules in your tasks/main.yml

@pedroluislopez

Copy link
Copy Markdown
Member Author

@pedroluislopez
It seems prometheus_exporter doesn't install any Debian packages, right? I don't see any apt modules in your tasks/main.yml

@hungpr0 It uses golang role to install prometheus exporter packages. So, you can install Debian packages in this way (you can find these examples in documentation):

prometheus_exporter__exporters:

 - name: 'nginx'
   private_port: '3113'
   public_port: '9113'
   apt_packages: 'prometheus-nginx-exporter'
       
 - name: 'mysqld'
   private_port: '3104'
   public_port: '9104'
   apt_packages: 'prometheus-mysqld-exporter'
   
 - name: 'mongodb'
   private_port: '3216'
   public_port: '9216'
   apt_packages: 'prometheus-mongodb-exporter'
   
prometheus_exporter__args:

 - name: 'nginx'
   options:
     - nginx.scrape-uri: 'https://{{ nginx_fqdn }}/nginx_status'
     
 - name: 'mongodb'
   options:
     - mongodb.uri: '{{ """mongodb://monitor:" +
                        lookup("password", secret + "/mongodb/" + ansible_fqdn + "/credentials/monitor/password") +
                        "@localhost:27017""" }}'

@hungpr0

hungpr0 commented Aug 5, 2021

Copy link
Copy Markdown
Contributor

@pedroluislopez

Thanks for your info. I missed to import golang role.
Just let you know that I had problem to start prometheus-apache-exporter due to different flags

@pedroluislopez
It seems prometheus_exporter doesn't install any Debian packages, right? I don't see any apt modules in your tasks/main.yml

@hungpr0 It uses golang role to install prometheus exporter packages. So, you can install Debian packages in this way (you can find these examples in documentation):

prometheus_exporter__exporters:

 - name: 'nginx'
   private_port: '3113'
   public_port: '9113'
   apt_packages: 'prometheus-nginx-exporter'
       
 - name: 'mysqld'
   private_port: '3104'
   public_port: '9104'
   apt_packages: 'prometheus-mysqld-exporter'
   
 - name: 'mongodb'
   private_port: '3216'
   public_port: '9216'
   apt_packages: 'prometheus-mongodb-exporter'
   
prometheus_exporter__args:

 - name: 'nginx'
   options:
     - nginx.scrape-uri: 'https://{{ nginx_fqdn }}/nginx_status'
     
 - name: 'mongodb'
   options:
     - mongodb.uri: '{{ """mongodb://monitor:" +
                        lookup("password", secret + "/mongodb/" + ansible_fqdn + "/credentials/monitor/password") +
                        "@localhost:27017""" }}'

@pedroluislopez

Thanks for your info. I missed to import golang role.
Just let you know that I had problem to start prometheus-apache-exporter due to different flags

https://github.com/Lusitaniae/apache_exporter
Apache-exporter uses

  • --telemetry.address instead of --web.listen-address
  • --telemetry.endpoint instead of --web.telemetry-path

@pedroluislopez

pedroluislopez commented Aug 8, 2021

Copy link
Copy Markdown
Member Author

@pedroluislopez

Thanks for your info. I missed to import golang role.
Just let you know that I had problem to start prometheus-apache-exporter due to different flags

@pedroluislopez
It seems prometheus_exporter doesn't install any Debian packages, right? I don't see any apt modules in your tasks/main.yml

@hungpr0 It uses golang role to install prometheus exporter packages. So, you can install Debian packages in this way (you can find these examples in documentation):

prometheus_exporter__exporters:

 - name: 'nginx'
   private_port: '3113'
   public_port: '9113'
   apt_packages: 'prometheus-nginx-exporter'
       
 - name: 'mysqld'
   private_port: '3104'
   public_port: '9104'
   apt_packages: 'prometheus-mysqld-exporter'
   
 - name: 'mongodb'
   private_port: '3216'
   public_port: '9216'
   apt_packages: 'prometheus-mongodb-exporter'
   
prometheus_exporter__args:

 - name: 'nginx'
   options:
     - nginx.scrape-uri: 'https://{{ nginx_fqdn }}/nginx_status'
     
 - name: 'mongodb'
   options:
     - mongodb.uri: '{{ """mongodb://monitor:" +
                        lookup("password", secret + "/mongodb/" + ansible_fqdn + "/credentials/monitor/password") +
                        "@localhost:27017""" }}'

@pedroluislopez

Thanks for your info. I missed to import golang role.
Just let you know that I had problem to start prometheus-apache-exporter due to different flags

https://github.com/Lusitaniae/apache_exporter
Apache-exporter uses

  • --telemetry.address instead of --web.listen-address
  • --telemetry.endpoint instead of --web.telemetry-path

@hungpr0

In this case, you have to drop common args and set args by each exporter. Like this:

prometheus_exporter__args:

  - name: 'common'
    state: 'absent'

  - name: 'node'
    options:
      - web.listen-address: '{{ ("localhost" if prometheus_exporter__pki|bool else prometheus_exporter__bind) + ":" +
                                (item.private_port if prometheus_exporter__pki|bool else
                                item.public_port) }}'

  - name: 'apache'
    options:
      - telemetry.address: '...'
      - telemetry.endpoint: '...'

@drybjed drybjed added this to the DebOps v3.2.0 milestone Aug 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new feature requests to add new functionality priority: medium tag: monitoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants