Monitoring: "No datapoints found" #831

smartm0use · 2023-01-27T13:50:38Z

smartm0use
Jan 27, 2023

I'm running Prometheus for monitoring electrs but going to http://localhost:9090/graph?g0.range_input=1h&g0.expr=index_height&g0.tab=0 (as suggested here) I get "No datapoints found" error.

What am I doing wrong?

romanz · 2023-01-27T15:23:41Z

romanz
Jan 27, 2023
Maintainer

It seems that Prometheus doesn't collect electrs' metrics - see https://github.com/romanz/electrs/blob/d128aef758280c7c38f4d89604c66a7e9f6f2135/doc/monitoring.md

0 replies

smartm0use · 2023-01-27T15:46:00Z

smartm0use
Jan 27, 2023
Author

It is what I have done (I also reported that bug in the instructions) but I still get that error.
I'm connecting by remote access using <node_onion_address>:4224 instead of localhost:4224.

Could be this the cause of that error?

2 replies

romanz Jan 27, 2023
Maintainer

I don't think... do you see other metrics that prometheus collects by default?
For example: http://localhost:9090/classic/graph?g0.range_input=1h&g0.expr=node_load1&g0.tab=0

romanz Jan 27, 2023
Maintainer

My bad, see #831 (reply in thread)

smartm0use · 2023-01-27T16:07:12Z

smartm0use
Jan 27, 2023
Author

It doesn't work, I get the same error.

Here is my configuration at the end of the file located in "/etc/prometheus/prometheus.yml":

scrape_configs:
  - job_name: electrs
    static_configs:
      - targets: ['<address>.onion:4224']

Any way to check if there are errors in establishing connection to electrs? First time I use Prometheus...

1 reply

romanz Jan 27, 2023
Maintainer

Did you restart prometheus (sudo systemctl restart prometheus)?

Could you please share your /etc/prometheus/prometheus.yml

romanz · 2023-01-27T16:21:32Z

romanz
Jan 27, 2023
Maintainer

Could you please try using localhost:4224 to see if it works?

0 replies

smartm0use · 2023-01-27T16:28:31Z

smartm0use
Jan 27, 2023
Author

Sorry, I was wrong, default metrics are working.

Yes, I did restart Prometheus.

Here is the Yml file:

# Sample config for Prometheus.

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'example'

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    scrape_timeout: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']

  - job_name: node
    # If prometheus-node-exporter is installed, grab stats about the local
    # machine by default.
    static_configs:
      - targets: ['localhost:9100']

  - job_name: electrs
    static_configs:
      - targets: ['<my-address>.onion:4224']

I can't try to use localhost:4224 because electrs in on another machine.

2 replies

romanz Jan 27, 2023
Maintainer

You were right - the '<my-address>.onion:4224' target won't probably work without using a Tor proxy.

romanz Jan 27, 2023
Maintainer

https://tor.stackexchange.com/questions/6939/configure-tor-as-proxy#6950

smartm0use · 2023-01-27T18:08:59Z

smartm0use
Jan 27, 2023
Author

I'm already connected under Tor. It is active and running...

0 replies

smartm0use · 2023-01-27T22:30:59Z

smartm0use
Jan 27, 2023
Author

Do you know any verbose option or setting to see what is the error?

1 reply

romanz Jan 28, 2023
Maintainer

You can set prometheus log level to debug at /etc/default/prometheus:

ARGS="--log.level=debug"

Then, restart prometheus:

$ sudo systemctl restart prometheus
$ journalctl -u prometheus | tail
Jan 28 10:01:34 home prometheus[8410]: ts=2023-01-28T08:01:34.365Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:01:49 home prometheus[8410]: ts=2023-01-28T08:01:49.364Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:02:04 home prometheus[8410]: ts=2023-01-28T08:02:04.364Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:02:19 home prometheus[8410]: ts=2023-01-28T08:02:19.365Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:02:34 home prometheus[8410]: ts=2023-01-28T08:02:34.365Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:02:49 home prometheus[8410]: ts=2023-01-28T08:02:49.364Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:03:04 home prometheus[8410]: ts=2023-01-28T08:03:04.364Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:03:19 home prometheus[8410]: ts=2023-01-28T08:03:19.364Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:03:34 home prometheus[8410]: ts=2023-01-28T08:03:34.365Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:03:49 home prometheus[8410]: ts=2023-01-28T08:03:49.364Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"

smartm0use · 2023-01-28T14:42:29Z

smartm0use
Jan 28, 2023
Author

It was helpfull:

caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://<my-address>.onion:4224/metrics msg="Scrape failed" err="Get \"http://<my-address>.onion:4224/metrics\": dial tcp: lookup <my-address>.onion: no such host"

It seems that is adding "http://" prefix to my Onion address...
According to this issue is it a limit of Prometheus?

5 replies

romanz Jan 28, 2023
Maintainer

So it seems... I'm not very familiar with Go, but maybe you can use something similar to https://stackoverflow.com/questions/43682909/connect-to-onion-websites-on-tor-using-python?

smartm0use Jan 30, 2023
Author

Sorry, I do not get it. How does this can help me?

romanz Feb 2, 2023
Maintainer

I think you should configure Prometheus to use Tor Proxy, maybe via proxy_url:
prometheus/prometheus@7810448

romanz Feb 2, 2023
Maintainer

If it doesn't work, maybe you can run Prometheus server on the same machine with electrs?

smartm0use Feb 2, 2023
Author

Sorry... how and where do I set proxy_url parameter?
I would avoid to run another server...

Monitoring: "No datapoints found" #831

Uh oh!

Uh oh!

smartm0use Jan 27, 2023

Replies: 8 comments · 11 replies

Uh oh!

romanz Jan 27, 2023 Maintainer

Uh oh!

smartm0use Jan 27, 2023 Author

Uh oh!

romanz Jan 27, 2023 Maintainer

Uh oh!

Uh oh!

romanz Jan 27, 2023 Maintainer

Uh oh!

smartm0use Jan 27, 2023 Author

Uh oh!

romanz Jan 27, 2023 Maintainer

Uh oh!

romanz Jan 27, 2023 Maintainer

Uh oh!

smartm0use Jan 27, 2023 Author

Uh oh!

romanz Jan 27, 2023 Maintainer

Uh oh!

romanz Jan 27, 2023 Maintainer

Uh oh!

Uh oh!

smartm0use Jan 27, 2023 Author

Uh oh!

smartm0use Jan 27, 2023 Author

Uh oh!

romanz Jan 28, 2023 Maintainer

Uh oh!

smartm0use Jan 28, 2023 Author

Uh oh!

romanz Jan 28, 2023 Maintainer

Uh oh!

smartm0use Jan 30, 2023 Author

Uh oh!

romanz Feb 2, 2023 Maintainer

Uh oh!

romanz Feb 2, 2023 Maintainer

Uh oh!

Uh oh!

smartm0use Feb 2, 2023 Author

smartm0use
Jan 27, 2023

Replies: 8 comments 11 replies

romanz
Jan 27, 2023
Maintainer

smartm0use
Jan 27, 2023
Author

romanz Jan 27, 2023
Maintainer

romanz Jan 27, 2023
Maintainer

smartm0use
Jan 27, 2023
Author

romanz Jan 27, 2023
Maintainer

romanz
Jan 27, 2023
Maintainer

smartm0use
Jan 27, 2023
Author

romanz Jan 27, 2023
Maintainer

romanz Jan 27, 2023
Maintainer

smartm0use
Jan 27, 2023
Author

smartm0use
Jan 27, 2023
Author

romanz Jan 28, 2023
Maintainer

smartm0use
Jan 28, 2023
Author

romanz Jan 28, 2023
Maintainer

smartm0use Jan 30, 2023
Author

romanz Feb 2, 2023
Maintainer

romanz Feb 2, 2023
Maintainer

smartm0use Feb 2, 2023
Author