Repository navigation
[Network Beaconing Identification] Add process related fields to beaconing transform - #18748
Conversation
|
Looks good. My suggestion is to make |
| 'source.ip': | ||
| terms: | ||
| field: source.ip | ||
| 'process.executable': |
There was a problem hiding this comment.
🟠 High pivot_transform/transform.yml:353
The 12 new group_by fields added in this PR are not defined in fields.yml, so the destination index ml_beaconing-1.6.0 will use dynamic mapping. Since the Painless scripts convert process.pid and process.parent.pid to strings via .toString(), Elasticsearch may map these as text instead of keyword, breaking downstream queries and detection rules that expect consistent field types. Consider adding explicit field definitions to packages/beaconing/elasticsearch/transform/pivot_transform/fields/fields.yml.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file packages/beaconing/elasticsearch/transform/pivot_transform/transform.yml around line 353:
The 12 new `group_by` fields added in this PR are not defined in `fields.yml`, so the destination index `ml_beaconing-1.6.0` will use dynamic mapping. Since the Painless scripts convert `process.pid` and `process.parent.pid` to strings via `.toString()`, Elasticsearch may map these as `text` instead of `keyword`, breaking downstream queries and detection rules that expect consistent field types. Consider adding explicit field definitions to `packages/beaconing/elasticsearch/transform/pivot_transform/fields/fields.yml`.
|
LGTM! Agree with the macroscopeapp, the field definitions also need updating |
|
Pinging @elastic/sec-applied-ml (Team:Security-Applied ML) |
| name: process.working_directory | ||
| - external: ecs | ||
| name: process.pid | ||
| - external: ecs |
There was a problem hiding this comment.
Have you tested what the output of using the ECS fields results in?
As mentioned the transform tostring() may go to text or keyword
But does using external: ecs automatically then handle the process to store them in long?
Just to be sure how it actually works, and where that translation is being done
There was a problem hiding this comment.
Thank you Susan for mentioning this. I've added a processor in the ingest pipeline that will correctly cast the process related 4 fields to correct ECS type.
💚 Build Succeeded
History
|
| if (!v.isEmpty()) { | ||
| if (ctx.process == null) { ctx.process = new HashMap(); } | ||
| if (ctx.process.parent == null) { ctx.process.parent = new HashMap(); } | ||
| ctx.process.parent.pid = Long.parseLong(v); |
There was a problem hiding this comment.
Is there any possibility of ctx.process_parent_pid missing and how would that be handled?
There was a problem hiding this comment.
These fields can be missing, and that is handled by the null and empty checks in the ingest pipeline. If the value is missing, the entire block (e.g. lines 26–33 for process.parent.pid) is skipped, so the field is never written to the output document, it is simply do not exist from the transform's output.
susan-shu-c
left a comment
There was a problem hiding this comment.
Thanks for addressing review/questions!
|
Package beaconing - 1.6.0 containing this change is available at https://epr.elastic.co/package/beaconing/1.6.0/ |
…oning transform (elastic#18748) * Add process related fields to beaconing transform * update transform * update changelog.yml * add fallback logic for process.executable * update fields.yml and add _dev folder * correctly cast the process related fields in ingest pipeline
Proposed commit message
Add process related fields to beaconing transform
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots
Transform and detection alerts preview