[VMware] Add more vSphere/ESXi logs by philippkahr · Pull Request #8122 · elastic/integrations

philippkahr · 2023-10-06T15:51:45Z

Hi,

multiple things:

Added a bunch more log lines to test. not only vcenter/vsphere also ESXI host related.
Adapted the original GROK parser to not drop events, instead just store them. This should be more in line with what we want to achieve with logs+.
Added parsing for login, logout events.
Added syslog parsing scripts from Arista
Added support for the user agent parser
Bumped the ECS version to 8.10.0
Parsing DNS events into appropriate DNS fields.
Parse the ESXI SSH login and logout message

Since this is a bit more of a change. i am not sure that 1.9.0 is enough of a version bump, or if we want to go 2.0.0

See if it's possible to differentiate between ESXI and vsphere logs for event.dataset or something like this. I don't want to introduce a new datastream with: logs-vsphere_esxi.log. Don't think that this is needed.
figure out how to deal with multiline. It's not a JSON, not real KV as well. Bit weird to parse. Maybe try to parse only important stuff such as eventTypeId to ECS.

<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: info hostd[67566312] [Originator@1244 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->    key = 161,
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->    chainId = -897703048,
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->    createdTime = "1970-01-01T00:00:00Z",
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->    userName = "",
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->    host = (vim.event.HostEventArgument) {
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->       name = "esxihost",
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->       host = 'vim.HostSystem:ha-host'
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->    },
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->    eventTypeId = "esx.problem.vmsyslogd.remote.failure",
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->    arguments = (vmodl.KeyAnyValue) [
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->       (vmodl.KeyAnyValue) {
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->          key = "1",
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->          value = "192.168.1.1:1234"
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->       }
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->    ],
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->    objectId = "ha-host",
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: -->    objectType = "vim.HostSystem",
<166>1 2023-09-18T16:07:55.298Z esxihost Hostd: --> }

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

elasticmachine · 2023-10-06T16:04:20Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-12-14T14:32:06.352+0000
Duration: 22 min 30 sec

Test stats 🧪

Test	Results
Failed	0
Passed	17
Skipped	0
Total	17

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-10-06T16:12:17Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`4/4`)	💚
Classes	100.0% (`4/4`)	💚
Methods	100.0% (`37/37`)	💚
Lines	94.521% (`276/292`)	❕
Conditionals	100.0% (`0/0`)	💚

…t the end of the line

philippkahr · 2023-10-11T11:53:00Z

/test

philippkahr · 2023-11-03T14:41:39Z

FYI I am on PTO from 4th November till 20th november.

ishleenk17 · 2023-11-30T07:05:42Z

@philippkahr : There are some files for infoblox Integration. Was that added by mistake ?

elasticmachine · 2023-12-12T10:29:21Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

devamanv · 2023-12-12T17:43:29Z

@philippkahr I see that are still some unanswered review comments that were marked resolved without any action or comment on them. Can you please revisit or address those? This will help us close the review on this PR and approve the changes faster. Thanks

Co-authored-by: Aman <38116245+devamanv@users.noreply.github.com>

philippkahr · 2023-12-12T20:54:52Z

@devamanv ~~I don't see any comments that I didn't address, can you point me to them please?~~ I think I get it now, all my comments had a pending tag. I had to go to files and submit my own review 😵‍💫

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

devamanv · 2023-12-13T10:10:37Z

@devamanv ~~I don't see any comments that I didn't address, can you point me to them please?~~ I think I get it now, all my comments had a pending tag. I had to go to files and submit my own review 😵‍💫

That's fine, just wanted to make sure we were tracking the progress on this one. The changes look good. I will review the changes again once all the CI checks have passed and complete my part of the review. Thanks for addressing the review comments.

philippkahr · 2023-12-13T12:04:14Z

@devamanv can you check the buildkite?


failed to solve: process "/bin/sh -c go get -u github.com/vmware/govmomi/vcsim@v0.25.0" did not complete successfully: exit code: 2
--
  | Error: error running package system tests: could not complete test run: could not setup service: could not boot up service using Docker Compose: running Docker Compose up command failed: exit status 17

I don't think that's on me. Since I didn't touch the govmomi go stuff.

devamanv · 2023-12-14T06:31:22Z

@philippkahr I have a hunch. Could you try to change the Go version to 1.17 in this file packages/vsphere/_dev/deploy/docker/docker-compose.yml and see if it fixes the issue? If it does, will try to explain the potential cause of this.

devamanv · 2023-12-14T08:16:07Z

So, a plausible explanation of what could have caused the checks to fail is as follows:
I tried downloading the package that is failing in the CI and noticed that it's trying to download the package google/uuid v1.5.0

> go get -u -v github.com/vmware/govmomi/vcsim@v0.25.0
go: downloading github.com/vmware/govmomi v0.25.0
go: downloading github.com/google/uuid v0.0.0-20170306145142-6a5e28554805
go: downloading github.com/google/uuid v1.5.0

This in turn threw the following error:

timeNow().UnixMilli undefined (type time.Time has no field or method UnixMilli)

Note that I was able to see this error by adding the -v flag

The uuid package recently added this change to support v7 of UUID. Finally, this implementation uses time.UnixMilli function, which was introduced in Go v1.17. So, the only option to fix this error was to upgrade the Go version.

devamanv · 2023-12-14T11:08:28Z

@philippkahr Now that we have updated the Go version, we might as well make another change to the dockerfile to replace the go get with go install.
From the official documentation:

Starting with Go 1.17, installing executables with go get is deprecated. go install may be used instead.

A similar change was done here in the vSphere module.

Co-authored-by: Aman <38116245+devamanv@users.noreply.github.com>

muthu-mps · 2023-12-14T14:31:50Z

/test

devamanv

LGTM!

elasticmachine · 2023-12-14T15:24:32Z

Package vsphere - 1.10.0 containing this change is available at https://epr.elastic.co/search?package=vsphere

large changes

2399873

philippkahr added enhancement New feature or request bugfix Pull request that fixes a bug issue labels Oct 6, 2023

philippkahr requested a review from a team as a code owner October 6, 2023 15:51

Added changelog

8e58cf3

philippkahr added 11 commits October 9, 2023 10:27

fixed a few things

d621fa5

Major rework. Sub pipelines for DNS and login

35daa96

Added more fields for security

bdb550e

Additional rework.

cf35835

Modified user extraction

ad1a632

fixed user agent parsing

df6cd13

Added test case for the grok pattern for the user agent

ecd2ae8

Moved event.kind: event, to the general pipeline

653a15e

There is also user_agent.original in the logs, that does not use )] a…

30628d9

…t the end of the line

Fixed appending event.action

56fd91b

Bump ECS to 8.10.0 and raise Kibana level

d671c67

philippkahr added 3 commits October 24, 2023 09:07

Added 2 more ecs

40ae8eb

Forgot to build the new readme

6fdb7bc

Added support for one more log message

a4c1907

philippkahr requested a review from a team as a code owner November 3, 2023 14:24

Found an additional log file and fixed event.outcome

59eddb7

lalit-satapathy requested review from ishleenk17 and muthu-mps November 27, 2023 09:41

ishleenk17 reviewed Nov 30, 2023

View reviewed changes

Comment thread packages/vsphere/data_stream/log/elasticsearch/ingest_pipeline/default.yml

Comment thread packages/vsphere/data_stream/log/elasticsearch/ingest_pipeline/default.yml Outdated

Comment thread packages/vsphere/data_stream/log/elasticsearch/ingest_pipeline/default.yml

ishleenk17 reviewed Nov 30, 2023

View reviewed changes

Comment thread packages/vsphere/data_stream/log/elasticsearch/ingest_pipeline/login.yml

remove license field

1c1e755

muthu-mps reviewed Dec 12, 2023

View reviewed changes

Comment thread packages/vsphere/manifest.yml Outdated

update kibana version constraint

0f5da26

shmsr reviewed Dec 12, 2023

View reviewed changes

Comment thread packages/vsphere/data_stream/log/elasticsearch/ingest_pipeline/default.yml

devamanv reviewed Dec 12, 2023

View reviewed changes

Comment thread packages/vsphere/changelog.yml Outdated

Update packages/vsphere/changelog.yml

225a9b4

Co-authored-by: Aman <38116245+devamanv@users.noreply.github.com>

muthu-mps reviewed Dec 13, 2023

View reviewed changes

Comment thread packages/vsphere/changelog.yml Outdated

Update packages/vsphere/changelog.yml

8e7ee76

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

bump go version

4bcdd51

devamanv changed the title ~~[VMWARE] vsphere logs~~ [VMware] vSphere logs Dec 14, 2023

devamanv changed the title ~~[VMware] vSphere logs~~ [VMware] Add more vSphere/ESXi logs Dec 14, 2023

philippkahr added 2 commits December 14, 2023 12:41

fixed API invocations thousand seperator

4cf04e9

Go install instead of go get

42c0857

devamanv reviewed Dec 14, 2023

View reviewed changes

Comment thread packages/vsphere/_dev/deploy/docker/Dockerfile Outdated

philippkahr and others added 2 commits December 14, 2023 13:36

Update packages/vsphere/_dev/deploy/docker/Dockerfile

f33814f

Co-authored-by: Aman <38116245+devamanv@users.noreply.github.com>

GOVMOMI version same as upstream beats

c130e5f

devamanv approved these changes Dec 14, 2023

View reviewed changes

philippkahr merged commit 4ccf4b9 into elastic:main Dec 14, 2023

philippkahr deleted the vmware-vsphere-grok branch December 14, 2023 15:12

andrewkroh added the Integration:vsphere VMware vSphere label Jul 22, 2024

Conversation

philippkahr commented Oct 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

elasticmachine commented Oct 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

elasticmachine commented Oct 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

philippkahr commented Oct 11, 2023

Uh oh!

philippkahr commented Nov 3, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ishleenk17 commented Nov 30, 2023

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Dec 12, 2023

🚀 Benchmarks report

Uh oh!

Uh oh!

Uh oh!

devamanv commented Dec 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philippkahr commented Dec 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

devamanv commented Dec 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philippkahr commented Dec 13, 2023

Uh oh!

devamanv commented Dec 14, 2023

Uh oh!

devamanv commented Dec 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devamanv commented Dec 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

muthu-mps commented Dec 14, 2023

Uh oh!

devamanv left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Dec 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

philippkahr commented Oct 6, 2023 •

edited

Loading

elasticmachine commented Oct 6, 2023 •

edited

Loading

elasticmachine commented Oct 6, 2023 •

edited

Loading

devamanv commented Dec 12, 2023 •

edited

Loading

philippkahr commented Dec 12, 2023 •

edited

Loading

devamanv commented Dec 13, 2023 •

edited

Loading

devamanv commented Dec 14, 2023 •

edited

Loading

devamanv commented Dec 14, 2023 •

edited

Loading