Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

elasticsearch input: ability to get base cluster health information without querying for indices health #3269

Merged
merged 2 commits into from
Oct 4, 2017

Conversation

meilke
Copy link
Contributor

@meilke meilke commented Sep 26, 2017

We found that the plugin generates a lot of data and also makes Elasticsearch work quite hard to generate the data (in some cases not necessary, if you are not interested in all the data).

New configurability:

  • ability to get base cluster health information without querying for indices health
## Set cluster_health to true when you want to also obtain cluster health stats
cluster_health = false

## Adjust cluster_health_level when you want to also obtain detailed health stats
## The options are
##  - indices (default)
##  - cluster
# cluster_health_level = "indices"

Required for all PRs:

  • Signed CLA.
  • Associated README.md updated.
  • Has appropriate unit tests.

@meilke
Copy link
Contributor Author

meilke commented Sep 29, 2017

Any thoughts?

@danielnelson
Copy link
Contributor

There is a bit of overlap with an existing pull request #2872, can you remove the indices_health code?

I feel that the url generation is too confusing, partially because of the current design. It needs to be made more straightforward and shouldn't cache the partial result individualStatsUrlPart onto the struct. Also can you explain why individualStatIncluded() is needed?

@meilke
Copy link
Contributor Author

meilke commented Oct 2, 2017

@danielnelson thanks for pointing out the existing PR situation regarding Elasticsearch. We desperately need the indices_health solution. We have a relatively big cluster and asking it for indices health takes upwards of 5 seconds and thus produces load we do not want. I am sure we can find a solution on how to fit things in nicely code-wise.

I did the individualStatIncluded approach to make the other method a little bit more readable but I am totally flexible on that. Regarding the caching: I must confess, I am still a little bit in the dark regarding processor life cycle. When would be a good hook to generate a concatenated string from the array? Because it is much preferred to have an array in the first place. Much nicer to read and understand in the config.

Other than that, there should not be too much overlap with the PR you mentioned, right?

And: What is the schedule regarding all these fixes for the Elasticsearch plugin?

@danielnelson
Copy link
Contributor

Actually, I think I confused indices_health (/_cluster/health?level=indices) with what the other PR calls indices_stats (/_all/_stats). However it would still be best if we make changes in separate pull requests.

We can talk about this more on the new PR but, one issue I have is with the dependency of boolean config variables on each other, when using boolean options they should be independant:

if e.ClusterHealth {
	url = s + "/_cluster/health"
	if e.IndicesHealth {

To avoid this confusion about when indices_health can be used I suggest something like:

cluster_health = true # (ship has sailed on this one)
cluster_health_level = ["indices"]

It will be a little trickier because we need to preserve backwards compatibility, so if the list is nil it should default to "indices", but if it is defined it will be none.

When would be a good hook to generate a concatenated string from the array? Because it is much preferred to have an array in the first place. Much nicer to read and understand in the config.

I agree this is how we want it in the config. I would start by just not caching it at all, it is not that much work in the grand scheme of things, if it ends up being a hotspot we can fix it later.

@meilke meilke changed the title elasticsearch input: more configurability elasticsearch input: ability to get base cluster health information without querying for indices health Oct 4, 2017
@meilke
Copy link
Contributor Author

meilke commented Oct 4, 2017

Thanks for the feedback, @danielnelson. I split this into this PR and #3304.

@danielnelson danielnelson merged commit 0bb3257 into influxdata:master Oct 4, 2017
@danielnelson danielnelson added this to the 1.5.0 milestone Oct 4, 2017
@danielnelson
Copy link
Contributor

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants