Skip to content

[SIEM] fix timeline/kql search disparity - #42843

Merged
stephmilovic merged 4 commits into
elastic:masterfrom
stephmilovic:fix-kql-timeline-disparity
Aug 8, 2019
Merged

stephmilovic merged 4 commits into
elastic:masterfrom
stephmilovic:fix-kql-timeline-disparity

Conversation

@stephmilovic

@stephmilovic stephmilovic commented Aug 7, 2019 •

Copy link
Copy Markdown
Contributor

Summary

This PR resolves this issue: #42549

The KQL query was getting appended to the timerange like this source.ip: 10.142.0.9 OR (dest.ip: 10.142.0.9 AND @timestamp in range) instead of to the timeline portion like this (source.ip: 10.142.0.9 OR dest.ip: 10.142.0.9) AND @timestamp in range. By moving the ) in the kuery string to the end of the KQL append, this fixes the issue.

Note: no more disparity in the number of events displayed in the lower left corner!
Screen Shot 2019-08-07 at 9 21 51 AM
Screen Shot 2019-08-07 at 9 22 03 AM

Checklist

Use strikethroughs to remove checklist items you don't feel are applicable to this PR.

For maintainers

@stephmilovic stephmilovic added Team:SIEM release_note:skip Skip the PR/issue when compiling release notes labels Aug 7, 2019
@elasticmachine

Copy link
Copy Markdown
Contributor

Pinging @elastic/siem

@elasticmachine

Copy link
Copy Markdown
Contributor

💚 Build Succeeded

@elasticmachine

Copy link
Copy Markdown
Contributor

💔 Build Failed

Comment thread x-pack/legacy/plugins/siem/public/components/timeline/helpers.tsx Outdated
@elasticmachine

Copy link
Copy Markdown
Contributor

💚 Build Succeeded

Comment thread x-pack/legacy/plugins/siem/public/components/timeline/helpers.test.tsx Outdated
@elasticmachine

Copy link
Copy Markdown
Contributor

💚 Build Succeeded

@andrew-goldstein

Copy link
Copy Markdown
Contributor

The following documents some of the ad hoc tests I ran against the PR branch in our development environment:

All tests were run with an absolute date range: August 5 12:00 AM - August 6 11:30 PM

user "bob" as a Filter

Filter KQL - and
user.name: "bob" n/a

Result: 5272 Events

user "bob" as KQL (AND mode)

Filter KQL - and
n/a user.name: "bob"

Result: 5272 Events

user "bob" as KQL (OR mode)

Filter KQL - or
n/a user.name: "bob"

Result: 5272 Events

user "bob" as both a Filter and KQL (AND mode) (should return the same results)

Filter KQL - and
user.name: "bob" user.name: "bob"

Result: 5272 Events

user "bob" as both a Filter and KQL (OR mode) (should return the same results)

Filter KQL - or
user.name: "bob" user.name: "bob"

Result: 5272 Events

user "bob" AND host "jessie" as Filters

Filter KQL - and
user.name: "bob" and host.name "jessie" n/a

Result: 5272 Events

user "bob" AND host "jessie" as KQL (AND mode)

Filter KQL - and
n/a user.name: "bob" and host.name: "jessie"

Result: 5272 Events

user "bob" AND host "jessie" as KQL (OR mode)

Filter KQL - or
n/a user.name: "bob" and host.name: "jessie"

Result: 5272 Events

user "bob" AND host "jessie" as both a Filter and KQL (AND mode) (should return the same results)

Filter KQL - and
user.name: "bob" and host.name: "jessie" user.name: "bob" and host.name: "jessie"

Result: 5272 Events

user "bob" AND host "jessie" as both a Filter and KQL (OR mode) (should return the same results)

Filter KQL - or
user.name: "bob" and host.name: "jessie" user.name: "bob" and host.name: "jessie"

Result: 5272 Events

user "bob" OR host "jessie" as Filters

Filter KQL - and
user.name: "bob" OR host.name: "jessie" n/a

Result: 5370 Events

user "bob" OR host "jessie" as KQL (AND mode)

Filter KQL - and
n/a user.name: "bob" OR host.name: "jessie"

Result: 5370 Events

user "bob" OR host "jessie" as KQL (OR mode)

Filter KQL - or
n/a user.name: "bob" OR host.name: "jessie"

Result: 5370 Events

user "bob" OR host "jessie" as both a Filter and KQL (AND mode) (should return the same results)

Filter KQL - and
user.name: "bob" OR host.name: "jessie" user.name: "bob" OR host.name: "jessie"

Result: 5370 Events

user "bob" OR host "jessie" as both a Filter and KQL (OR mode) (should return the same results)

Filter KQL - or
user.name: "bob" OR host.name: "jessie" user.name: "bob" OR host.name: "jessie"

Result: 5370 Events

user "bob" OR host "jessie" split between Filter and KQL (OR mode)

Filter KQL - or
user.name: "bob" host.name: "jessie"

Result: 5370 Events

user "bob" OR host "jessie" split between Filter and KQL (AND mode)

Filter KQL - and
user.name: "bob" host.name: "jessie"

Result: 5270 Events <-- NOTE: similar queries yield 5272 events

user "bob" AND host "jessie" or user "bob" split between Filter and KQL (AND mode)

Filter KQL - and
user.name: "bob" host.name: "jessie" OR user.name: "bob"

Result: 5272 Events

user "bob" AND host "jessie" or user "bob" split between Filter and KQL (OR mode)

Filter KQL - or
user.name: "bob" host.name: "jessie" OR user.name: "bob"

Result: 5370 Events

user.name exists as a Filter

Filter KQL - and
user.name: exists n/a

Result: 9775066 Events

user.name exists as KQL (AND mode)

Filter KQL - and
n/a user.name: *

Result: 9775066 Events

user.name exists as KQL (OR mode)

Filter KQL - and
n/a user.name: *

Result: 9775066 Events

user "bob" AND user.name exists, split between Filter and KQL (AND mode)

Filter KQL - and
user.name: bob user.name: *

Result: 5272 Events

user "bob" OR user.name exists, split between Filter and KQL (OR mode)

Filter KQL - or
user.name: bob user.name: *

Result: 9775066 Events

host "jessie" OR user "bob" AND user.name exists, split between Filter and KQL (AND mode)

Filter KQL - and
host.name: "jessie" OR user.name: "bob" user.name: *

Result: 5272 Events

host "jessie" OR user "bob" OR user.name exists, split between Filter and KQL (OR mode)

Filter KQL - or
host.name: "jessie" OR user.name: "bob" user.name: *

Result: 9775066 Events

@andrew-goldstein andrew-goldstein left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🦅 👀 Thanks @stephmilovic for catching this disparity, and for the fix in this PR! In addition to the fix itself, the removal of the extra ( )s in the generated KQL query is icing on the 🍰.

I ran some ad hoc tests against the development environment and added some of the results as a comment to this PR.

LGTM 🙏 🦅

@spong spong left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked out, tested locally, and also performed a code review. Gave the Query Builder + KQL AND/OR's quite a bit of testing and all event counts + queries looked good! Did come across this separate escaping issue in testing (#42866), but unrelated.

Looooks good to meeeee! Great job with these changes @stephmilovic! 😃🎉

@stephmilovic
stephmilovic merged commit 3a310b0 into elastic:master Aug 8, 2019
@stephmilovic
stephmilovic deleted the fix-kql-timeline-disparity branch August 8, 2019 13:21
stephmilovic added a commit to stephmilovic/kibana that referenced this pull request Aug 8, 2019
@tsg tsg added release_note:fix v7.4.0 and removed release_note:fix release_note:skip Skip the PR/issue when compiling release notes labels Aug 19, 2019
@tsg

tsg commented Aug 19, 2019

Copy link
Copy Markdown
Contributor

@stephmilovic Generally bug fixes should have release_note:fix so that they make it into the release notes. Let me know if you had a particular reason to release_note:skip this one.

patrykkopycinski pushed a commit to patrykkopycinski/kibana that referenced this pull request May 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants