[Ceph] Add Integration Package with Cluster Health data stream by harnish-crest-data · Pull Request #4963 · elastic/integrations

harnish-crest-data · 2023-01-10T13:54:03Z

Enhancement

What does this PR do?

Added 1 data stream (Cluster Health metrics).
Added data collection logic for the data streams.
Added the ingest pipeline for the data streams.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yaml files.
Added system test cases for the data stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic-package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/ceph directory.
Run the following command to run tests.
elastic-package test

Related issues

elasticmachine · 2023-01-10T13:57:27Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-02-06T12:45:52.918+0000
Duration: 15 min 58 sec

Test stats 🧪

Test	Results
Failed	0
Passed	10
Skipped	0
Total	10

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

kush-elastic · 2023-01-11T04:24:16Z

/test

elasticmachine · 2023-01-11T04:38:21Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`2/2`)	💚
Files	100.0% (`2/2`)	💚
Classes	100.0% (`2/2`)	💚
Methods	100.0% (`16/16`)	💚
Lines	91.489% (`86/94`)	👍
Conditionals	100.0% (`0/0`)	💚

kush-elastic

LGTM!

agithomas · 2023-01-17T04:52:57Z

I do not see dashboard specific files? Doesn't the datastream comes with visualisation ?

harnish-crest-data · 2023-01-17T05:11:59Z

I do not see dashboard specific files? Doesn't the datastream comes with visualisation ?

The data stream does not have any visualization. Hence we haven't any dashboard specific files.

agithomas · 2023-01-17T09:15:57Z

Please use the actual docker image for creating system testcases. Reference : https://quay.io/repository/ceph/ceph?tab=tags, https://hub.docker.com/u/ceph, https://docs.ceph.com/en/latest/install/containers/

harnish-crest-data · 2023-01-19T04:32:23Z

Please use the actual docker image for creating system testcases. Reference : https://quay.io/repository/ceph/ceph?tab=tags, https://hub.docker.com/u/ceph, https://docs.ceph.com/en/latest/install/containers/

We already have gone through the ceph images which you have mentioned while doing system test. Also we have gone through the https://hub.docker.com/r/ceph/daemon image.

We are facing this issue while doing docker-compose up,

We are facing this similar issues when performing ceph operations in the docker image.

Also we have referred the ceph metricbeat module how they are doing system tests.

So we have performed similar steps with our system test. But we are facing the issue related to API Secret Key.
We need to pass the API User and API Secret Key, while configuring the ceph integration. (Which we already mentioned how to get this details in our README.md)
For the API User, the environment is set in that way like we have static value for API User (demo).
But for the API Secret Key, the system is having dynamic value. (Generated by Ceph environment)
So for that we need to set the API Secret Key to environment variable and pass it while doing system test.
But somehow we are not able to do this while doing system test.

Hence we have to perform the system test using the mock server.

agithomas · 2023-01-30T06:34:09Z

Please use the actual docker image for creating system testcases. Reference : https://quay.io/repository/ceph/ceph?tab=tags, https://hub.docker.com/u/ceph, https://docs.ceph.com/en/latest/install/containers/

Please use the actual docker image for creating system testcases. Reference : https://quay.io/repository/ceph/ceph?tab=tags, https://hub.docker.com/u/ceph, https://docs.ceph.com/en/latest/install/containers/

We already have gone through the ceph images which you have mentioned while doing system test. Also we have gone through the https://hub.docker.com/r/ceph/daemon image.

We are facing this issue while doing docker-compose up, We are facing this similar issues when performing ceph operations in the docker image.

Also we have referred the ceph metricbeat module how they are doing system tests.

So we have performed similar steps with our system test. But we are facing the issue related to API Secret Key. We need to pass the API User and API Secret Key, while configuring the ceph integration. (Which we already mentioned how to get this details in our README.md) For the API User, the environment is set in that way like we have static value for API User (demo). But for the API Secret Key, the system is having dynamic value. (Generated by Ceph environment) So for that we need to set the API Secret Key to environment variable and pass it while doing system test. But somehow we are not able to do this while doing system test.

Hence we have to perform the system test using the mock server.

@harnish-elastic , Can you please create an issue in the elastic-package repo explaining the requirement

harnish-crest-data · 2023-02-01T07:25:05Z

Please use the actual docker image for creating system testcases. Reference : https://quay.io/repository/ceph/ceph?tab=tags, https://hub.docker.com/u/ceph, https://docs.ceph.com/en/latest/install/containers/

Please use the actual docker image for creating system testcases. Reference : https://quay.io/repository/ceph/ceph?tab=tags, https://hub.docker.com/u/ceph, https://docs.ceph.com/en/latest/install/containers/

We already have gone through the ceph images which you have mentioned while doing system test. Also we have gone through the https://hub.docker.com/r/ceph/daemon image.
We are facing this issue while doing docker-compose up, We are facing this similar issues when performing ceph operations in the docker image.
Also we have referred the ceph metricbeat module how they are doing system tests.
So we have performed similar steps with our system test. But we are facing the issue related to API Secret Key. We need to pass the API User and API Secret Key, while configuring the ceph integration. (Which we already mentioned how to get this details in our README.md) For the API User, the environment is set in that way like we have static value for API User (demo). But for the API Secret Key, the system is having dynamic value. (Generated by Ceph environment) So for that we need to set the API Secret Key to environment variable and pass it while doing system test. But somehow we are not able to do this while doing system test.
Hence we have to perform the system test using the mock server.

@harnish-elastic , Can you please create an issue in the elastic-package repo explaining the requirement

Raised issue in elastic-package repo explaining requirement. Link: elastic/elastic-package#1126

agithomas · 2023-02-02T05:35:49Z

metric_type mapping is missing. Please add them.

harnish-crest-data · 2023-02-02T13:48:04Z

metric_type mapping is missing. Please add them.

Updated, thanks!

agithomas

Please check the comments

agithomas

LGTM

…into package-ceph_cluster_health

elasticmachine · 2023-02-06T13:40:26Z

Package ceph - 0.2.0 containing this change is available at https://epr.elastic.co/search?package=ceph

…ic#4963) * Add Ceph integration package with cluster_health data stream * update changelog.yml file * address review comments * minor changes in readme * elastic-package check

Add Ceph integration package with cluster_health data stream

5d14320

update changelog.yml file

c39ad79

kush-elastic requested review from agithomas and kush-elastic January 11, 2023 04:25

kush-elastic assigned harnish-crest-data Jan 11, 2023

kush-elastic added enhancement New feature or request Integration:ceph Ceph Team:Service-Integrations Label for the Observability Service Integrations team labels Jan 11, 2023

kush-elastic marked this pull request as ready for review January 11, 2023 04:39

kush-elastic approved these changes Jan 11, 2023

View reviewed changes

kush-elastic requested review from SubhrataK and lalit-satapathy January 11, 2023 05:32

This was referenced Jan 16, 2023

[Ceph] Add Integration Package with Cluster Disk data stream #5011

Merged

[Ceph] Add Integration Package with OSD Performance data stream #4930

Merged

This was referenced Jan 20, 2023

[Ceph] Add Integration Package with Pool Disk data stream #5070

Merged

[Ceph] Add Integration Package with OSD Pool Stats data stream #5128

Merged

harnish-crest-data mentioned this pull request Jan 31, 2023

[Ceph] Add Integration Package with dashboard #5152

Merged

4 tasks done

harnish-crest-data mentioned this pull request Feb 2, 2023

[Ceph] Add Integration Package with OSD Tree data stream #5174

Merged

4 tasks done

harnish-crest-data added 2 commits February 2, 2023 18:56

address review comments

b8abbaa

minor changes in readme

1f36672

elastic-package check

5f39ba0

agithomas reviewed Feb 4, 2023

View reviewed changes

Comment thread packages/ceph/data_stream/cluster_health/fields/fields.yml

agithomas reviewed Feb 4, 2023

View reviewed changes

agithomas approved these changes Feb 6, 2023

View reviewed changes

harnish-crest-data added 2 commits February 6, 2023 18:01

Merge branch 'main' of https://github.com/harnish-elastic/integrations …

a95fdf1

…into package-ceph_cluster_health

Merge branch 'main' of https://github.com/harnish-elastic/integrations …

caab790

…into package-ceph_cluster_health

harnish-crest-data requested a review from a team as a code owner February 6, 2023 12:45

kush-elastic merged commit 0c55815 into elastic:main Feb 6, 2023

harnish-crest-data mentioned this pull request Feb 8, 2023

[Ceph] Add Integration Package with Cluster Status data stream #5216

Merged

4 tasks done

Conversation

harnish-crest-data commented Jan 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Jan 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

kush-elastic commented Jan 11, 2023

Uh oh!

elasticmachine commented Jan 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

kush-elastic left a comment

Choose a reason for hiding this comment

Uh oh!

agithomas commented Jan 17, 2023

Uh oh!

harnish-crest-data commented Jan 17, 2023

Uh oh!

agithomas commented Jan 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

harnish-crest-data commented Jan 19, 2023

Uh oh!

agithomas commented Jan 30, 2023

Uh oh!

harnish-crest-data commented Feb 1, 2023

Uh oh!

agithomas commented Feb 2, 2023

Uh oh!

harnish-crest-data commented Feb 2, 2023

Uh oh!

Uh oh!

agithomas left a comment

Choose a reason for hiding this comment

Uh oh!

agithomas left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Feb 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

harnish-crest-data commented Jan 10, 2023 •

edited

Loading

elasticmachine commented Jan 10, 2023 •

edited

Loading

elasticmachine commented Jan 11, 2023 •

edited

Loading

agithomas commented Jan 17, 2023 •

edited

Loading