Skip to content

Update http and url ECS fields #1813

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jalvz merged 1 commit into from
Jan 24, 2019
Merged

Update http and url ECS fields #1813

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
191 changes: 96 additions & 95 deletions _meta/ecs-migration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,12 @@
#- from: beat.version
# to: observer.version
#

# field in ECS, but with a different meaning
- from: context.service.agent.name
to: agent.name
#

# field in ECS, but with a different meaning
- from: context.service.agent.version
to: agent.version

Expand All @@ -35,42 +38,50 @@
- from: context.system.platform
to: host.os.platform

#- from: context.request.method
# to: http.request.method
#
#- from: context.request.http_version
# to: http.version
#
#- from: context.request.body
# to: http.request.body
# index: false
#
## Not in ECS
#- from: context.request.cookies
# to: http.request.headers.cookies.parsed
# index: false
#
## Not in ECS
#- from: context.request.headers.cookie
# to: http.request.headers.cookies.original
# index: false
#
#- from: context.request.headers.user-agent
# to: http.request.headers.user-agent
# index: false
#
#- from: context.request.headers.content-type
# to: http.request.headers.content-type
# index: false
#
#- from: context.request.env
# to: http.request.env
# index: false
#
#- from: context.request.socket
# to: http.request.socket
# index: false
#
# from upper case to lower case
- from: context.request.method
to: http.request.method

- from: context.request.http_version
to: http.version

# conflicts with ECS, we can't align on name because we don't index it
- from: context.request.body
to: http.request.body.original
index: false

# not in ECS
- from: context.request.cookies
to: http.request.headers.cookies.parsed
index: false

# not in ECS
- from: context.request.headers.cookie
to: http.request.headers.cookies.original
index: false

# not in ECS
# `original` for consistency with root user_agent
# dashed for consistency with other headers
- from: context.request.headers.user-agent
to: http.request.headers.user-agent.original

@webmat webmat Jan 22, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually user_agent is in ECS: https://github.com/elastic/ecs#user_agent

@jalvz jalvz Jan 22, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the reason for that? the actual header name is User-Agent

@graphaelli graphaelli Jan 22, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should stay as user-agent since it's nested under http.request.headers but I'd also be ok to switch if everyone else prefers that

@webmat webmat Jan 22, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In ECS it's at the top level, so not really a conflict with how you normalize header names as field names (if you decide to move it at the top level).

If you decide to leave it nested here because you prefer to list it as one of the header entries, then I'd opt to leave as is, to keep the consistent usage of '-' with other headers.

My actual suggestion would be to move this to ECS, however (and I thought I had seen another entry in this file to do so).

Moving this field to the ECS name will ensure the user agents recorded by APM can be found easily by any integration or person that supports the common schema, without having to create an exception for APM. That's the whole point of ECS, after all :-)

@jalvz jalvz Jan 23, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, as you noted there is another entry in this file for that. We have 2 user agents: the one in the request to the apm-server (which is moved to the ECS field at the root) and this one, which is in a request possibly captured by an APM agent.

@webmat webmat Jan 23, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these present in the same event at the same time? If not, then both should break down the user agent in the ECS field, user_agent at the top level.

If they can both be present in the same event, then that's a more complicated question :-)

@jalvz jalvz Jan 24, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they are present at the same time.

index: false
Comment thread
simitt marked this conversation as resolved.

# not in ECS
- from: context.request.headers.content-type
to: http.request.headers.content-type
Comment thread
simitt marked this conversation as resolved.
index: false

# not in ECS
- from: context.request.env
to: http.request.env
index: false

@simitt simitt Jan 21, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not defined in ECS

@simitt simitt Jan 21, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The standard in ECS is that if the name does not end with text then it is a keyword. This is an object, so not sure how to bring this into ECS convention.

@jalvz jalvz Jan 22, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@graphaelli do you have any opinion on this?

@webmat webmat Jan 22, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The recommendation is to have all fields as keyword. If you need text datatype, you should have it as a multi-field. This is the reverse of the ES default.

Since this field is not in ECS, you're free to do what you want here.

@webmat webmat Jan 22, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And about things that aren't in ECS (like http.request.env), they're not automatically a problem. If we don't add it to ECS, there will never be a conflict. And in this case, it's perfectly fine for you to define this field there.

http.request.env is the variables defined in the application's context? If that's the case, I can guarantee it'll never be in ECS (as it has nothing to do with the HTTP protocol), and would not named like this (as we try to use full words as much as possible).

So no conflict here 👍


# not in ECS
- from: context.request.socket
to: http.request.socket
index: false
Comment thread
simitt marked this conversation as resolved.

- from: context.process.pid
to: process.pid

Expand Down Expand Up @@ -117,53 +128,48 @@
- from: context.service.version
to: service.version

#- from: context.request.url.full
# to: url.full
- from: context.request.url.full
to: url.full

#
#- from: context.request.url.hash
# to: url.fragment
- from: context.request.url.hash
to: url.fragment

#
#- from: context.request.url.hostname
# to: url.domain
- from: context.request.url.hostname
to: url.domain

#
#- from: context.request.url.pathname
# to: url.path
- from: context.request.url.pathname
to: url.path

#
#- from: context.request.url.port
# to: url.port
# from string or number to number
- from: context.request.url.port
to: url.port

#
#- from: context.request.url.raw
# to: url.original
# index: false
#
#- from: context.request.url.search
# to: url.query
- from: context.request.url.raw
to: url.original
index: false

#
#- from: context.request.url.protocol
# to: url.scheme
- from: context.request.url.search
to: url.query

#
#- from: context.response.finished
# to: http.response.finished
- from: context.request.url.protocol
to: url.scheme

#
#- from: context.response.status_code
# to: http.response.status_code
# not in ECS
- from: context.response.finished
Comment thread
simitt marked this conversation as resolved.
to: http.response.finished

#
#- from: context.response.headers.content-type
# to: http.response.headers.content-type
# index: false
#
#- from: context.response.headers_sent
# to: http.response.headers.sent
# index: false
- from: context.response.status_code
to: http.response.status_code

# not in ECS
- from: context.response.headers.content-type
to: http.response.headers.content-type
index: false

# not in ECS
- from: context.response.headers_sent
to: http.response.headers_sent
index: false

# not in ECS
- from: context.custom
Expand All @@ -185,30 +191,25 @@
to: context.db.content
index: false

# # not in ECS
#- from: context.db.user
# to: context.db.user
# index: false
#
#- from: context.user.email
# to: user.email
# not in ECS
- from: context.db.user
to: context.db.user

@simitt simitt Jan 21, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this kept under context while everything else is moved outside?

@webmat webmat Jan 22, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it stays there, just remove the entry. This file is for what's being migrated :-)

index: false

#
#- from: context.user.id
# to: user.id
- from: context.user.email
to: user.email

#
#- from: context.user.username
# to: user.name
- from: context.user.id
to: user.id

#
#- from: context.user.ip
# to: client.ip
- from: context.user.username
to: user.name

#
#- from: context.user.user-agent
# to: user_agent.original.text
- from: context.user.ip
to: client.ip

- from: context.user.user-agent
to: user_agent.original

#
- from: listening
to: observer.listening
Loading