Skip to content

[Network Beaconing Identification] Add process related fields to beaconing transform #18748

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sodhikirti07 merged 6 commits into from
May 5, 2026
Merged

[Network Beaconing Identification] Add process related fields to beaconing transform #18748

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
67003ff
Add process related fields to beaconing transform
sodhikirti07 Apr 30, 2026
db63a0f
update transform
sodhikirti07 Apr 30, 2026
c28613d
update changelog.yml
sodhikirti07 Apr 30, 2026
ab69419
add fallback logic for process.executable
sodhikirti07 Apr 30, 2026
38ec571
update fields.yml and add _dev folder
sodhikirti07 Apr 30, 2026
8f3d5a1
correctly cast the process related fields in ingest pipeline
sodhikirti07 May 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions packages/beaconing/_dev/build/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
dependencies:
ecs:
reference: "git@v8.17.0"
5 changes: 5 additions & 0 deletions packages/beaconing/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
- version: "1.6.0"
changes:
- description: Update beaconing transform to add process related fields
type: enhancement
link: https://github.com/elastic/integrations/pull/18748
Comment thread
macroscopeapp[bot] marked this conversation as resolved.
- version: "1.5.4"
changes:
- description: Readme improvement
Expand Down
49 changes: 48 additions & 1 deletion packages/beaconing/elasticsearch/ingest_pipeline/ml_beaconing_ingest_pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,51 @@ processors:
formats:
- UNIX
- ISO8601
target_field: event.ingested
target_field: event.ingested
- script:
description: Casts flat underscore process fields in group-by to ECS nested types.
lang: painless
source: |
if (ctx.process_pid != null) {
String v = ctx.process_pid.toString();
if (!v.isEmpty()) {
if (ctx.process == null) { ctx.process = new HashMap(); }
ctx.process.pid = Long.parseLong(v);
}
}
if (ctx.process_parent_pid != null) {
String v = ctx.process_parent_pid.toString();
if (!v.isEmpty()) {
if (ctx.process == null) { ctx.process = new HashMap(); }
if (ctx.process.parent == null) { ctx.process.parent = new HashMap(); }
ctx.process.parent.pid = Long.parseLong(v);

@susan-shu-c susan-shu-c May 1, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any possibility of ctx.process_parent_pid missing and how would that be handled?

@sodhikirti07 sodhikirti07 May 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These fields can be missing, and that is handled by the null and empty checks in the ingest pipeline. If the value is missing, the entire block (e.g. lines 26–33 for process.parent.pid) is skipped, so the field is never written to the output document, it is simply do not exist from the transform's output.

}
}
if (ctx.process_code_signature_exists != null) {
String v = ctx.process_code_signature_exists.toString();
if (!v.isEmpty()) {
if (ctx.process == null) { ctx.process = new HashMap(); }
if (ctx.process.code_signature == null) { ctx.process.code_signature = new HashMap(); }
ctx.process.code_signature.exists = Boolean.parseBoolean(v);
}
}
if (ctx.process_code_signature_trusted != null) {
String v = ctx.process_code_signature_trusted.toString();
if (!v.isEmpty()) {
if (ctx.process == null) { ctx.process = new HashMap(); }
if (ctx.process.code_signature == null) { ctx.process.code_signature = new HashMap(); }
ctx.process.code_signature.trusted = Boolean.parseBoolean(v);
}
}
- remove:
field: process_pid
ignore_missing: true
- remove:
field: process_parent_pid
ignore_missing: true
- remove:
field: process_code_signature_exists
ignore_missing: true
- remove:
field: process_code_signature_trusted
ignore_missing: true
44 changes: 34 additions & 10 deletions packages/beaconing/elasticsearch/transform/pivot_transform/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,33 @@
- name: host.name
type: keyword
- name: process.name
type: keyword
- name: source.ip
type: ip
- external: ecs
name: host.name
- external: ecs
name: process.name
- external: ecs
name: source.ip
- external: ecs
name: process.executable
- external: ecs
name: process.parent.name
- external: ecs
name: process.command_line
- external: ecs
name: process.parent.command_line
- external: ecs
name: process.code_signature.exists
- external: ecs
name: process.code_signature.trusted
- external: ecs
name: process.code_signature.subject_name
- external: ecs
name: process.hash.sha256
- external: ecs
name: process.parent.hash.sha256
- external: ecs
name: process.working_directory
- external: ecs
name: process.pid
- external: ecs

@susan-shu-c susan-shu-c May 1, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you tested what the output of using the ECS fields results in?
As mentioned the transform tostring() may go to text or keyword
But does using external: ecs automatically then handle the process to store them in long?
Just to be sure how it actually works, and where that translation is being done

@sodhikirti07 sodhikirti07 May 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you Susan for mentioning this. I've added a processor in the ingest pipeline that will correctly cast the process related 4 fields to correct ECS type.

name: process.parent.pid
- name: beacon_stats.autocovariance
type: float
- name: beacon_stats.beaconing_score
Expand Down Expand Up @@ -44,7 +68,7 @@
type: float
- name: beacon_stats.variance_counts
type: float
- name: '@timestamp'
type: date
- name: 'event.ingested'
type: date
- external: ecs
name: '@timestamp'
- external: ecs
name: event.ingested
66 changes: 63 additions & 3 deletions packages/beaconing/elasticsearch/transform/pivot_transform/transform.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
dest:
index: ml_beaconing-1.5.4
pipeline: 1.5.4-ml_beaconing_ingest_pipeline
index: ml_beaconing-1.6.0
pipeline: 1.6.0-ml_beaconing_ingest_pipeline
aliases:
- alias: ml_beaconing.latest
move_on_creation: true
Expand Down Expand Up @@ -350,6 +350,66 @@ pivot:
'source.ip':
terms:
field: source.ip
'process.executable':

@macroscopeapp macroscopeapp Bot Apr 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High pivot_transform/transform.yml:353

The 12 new group_by fields added in this PR are not defined in fields.yml, so the destination index ml_beaconing-1.6.0 will use dynamic mapping. Since the Painless scripts convert process.pid and process.parent.pid to strings via .toString(), Elasticsearch may map these as text instead of keyword, breaking downstream queries and detection rules that expect consistent field types. Consider adding explicit field definitions to packages/beaconing/elasticsearch/transform/pivot_transform/fields/fields.yml.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent: 
In file packages/beaconing/elasticsearch/transform/pivot_transform/transform.yml around line 353:

The 12 new `group_by` fields added in this PR are not defined in `fields.yml`, so the destination index `ml_beaconing-1.6.0` will use dynamic mapping. Since the Painless scripts convert `process.pid` and `process.parent.pid` to strings via `.toString()`, Elasticsearch may map these as `text` instead of `keyword`, breaking downstream queries and detection rules that expect consistent field types. Consider adding explicit field definitions to `packages/beaconing/elasticsearch/transform/pivot_transform/fields/fields.yml`.

terms:
script:
source: "return doc.containsKey('process.executable') && !doc['process.executable'].isEmpty() ? doc['process.executable'].value : ''"
lang: painless
'process.parent.name':
terms:
script:
source: "return doc.containsKey('process.parent.name') && !doc['process.parent.name'].isEmpty() ? doc['process.parent.name'].value : ''"
lang: painless
'process.command_line':
terms:
script:
source: "return doc.containsKey('process.command_line') && !doc['process.command_line'].isEmpty() ? doc['process.command_line'].value : ''"
lang: painless
'process.parent.command_line':
terms:
script:
source: "return doc.containsKey('process.parent.command_line') && !doc['process.parent.command_line'].isEmpty() ? doc['process.parent.command_line'].value : ''"
lang: painless
'process_code_signature_exists':
terms:
script:
source: "return doc.containsKey('process.code_signature.exists') && !doc['process.code_signature.exists'].isEmpty() ? doc['process.code_signature.exists'].value.toString() : ''"
lang: painless
'process_code_signature_trusted':
terms:
script:
source: "return doc.containsKey('process.code_signature.trusted') && !doc['process.code_signature.trusted'].isEmpty() ? doc['process.code_signature.trusted'].value.toString() : ''"
lang: painless
'process.code_signature.subject_name':
terms:
script:
source: "return doc.containsKey('process.code_signature.subject_name') && !doc['process.code_signature.subject_name'].isEmpty() ? doc['process.code_signature.subject_name'].value : ''"
lang: painless
'process.hash.sha256':
terms:
script:
source: "return doc.containsKey('process.hash.sha256') && !doc['process.hash.sha256'].isEmpty() ? doc['process.hash.sha256'].value : ''"
lang: painless
'process.parent.hash.sha256':
terms:
script:
source: "return doc.containsKey('process.parent.hash.sha256') && !doc['process.parent.hash.sha256'].isEmpty() ? doc['process.parent.hash.sha256'].value : ''"
lang: painless
'process.working_directory':
terms:
script:
source: "return doc.containsKey('process.working_directory') && !doc['process.working_directory'].isEmpty() ? doc['process.working_directory'].value : ''"
lang: painless
'process_pid':
terms:
script:
source: "return doc.containsKey('process.pid') && !doc['process.pid'].isEmpty() ? doc['process.pid'].value.toString() : ''"
lang: painless
'process_parent_pid':
terms:
script:
source: "return doc.containsKey('process.parent.pid') && !doc['process.parent.pid'].isEmpty() ? doc['process.parent.pid'].value.toString() : ''"
lang: painless
source:
index: logs-*
query:
Expand Down Expand Up @@ -394,5 +454,5 @@ sync:
delay: 120s
field: "@timestamp"
_meta:
fleet_transform_version: 1.5.4
fleet_transform_version: 1.6.0
run_as_kibana_system: false
2 changes: 1 addition & 1 deletion packages/beaconing/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 3.0.0
name: beaconing
title: "Network Beaconing Identification"
version: 1.5.4
version: 1.6.0
source:
license: "Elastic-2.0"
description: "Package to identify beaconing activity in your network events."
Expand Down
Loading