Skip to content

Seemingly spurious health check logs #6

@blakehawkins

Description

@blakehawkins

Arq's default health check interval is 3600 seconds (https://github.com/samuelcolvin/arq/blob/9109c2e59d2b13fa59d246da03d19d7844a6fa19/arq/worker.py#LL208C30-L208C30)

This causes spurious warn logs from time to time for me, which say:

'WARNING::arq.prometheus:159    [arq_prometheus] Health key could not be read, value is `None`.
Possible causes:
- health key has not been initialized by the worker yet
- `health_check_key` or `queue_name` settings may be wrong

This might be caused by an issue with redis persistence in my cluster, but in any case this led me to realise that arq-prometheus relies on this health check frequency for producing up-to-date metrics

Recommend adding health_check_interval to the Readme for arq worker settings.

Here is the patch that I used to resolve the warning spam:

            # Run arq health checks at double the frequency of our arq-prometheus monitoring:
            health_check_interval=datetime.timedelta(seconds=7.5),

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions