Skip to content

[ti_misp] Add agentless deployment - #19702

Merged
moxarth-rathod merged 5 commits into
elastic:mainfrom
moxarth-rathod:ti_misp-agentless
Jun 29, 2026
Merged

[ti_misp] Add agentless deployment#19702
moxarth-rathod merged 5 commits into
elastic:mainfrom
moxarth-rathod:ti_misp-agentless

Conversation

@moxarth-rathod

@moxarth-rathod moxarth-rathod commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Proposed commit message

ti_misp: add agentless support and update kibana version constraint

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

  • Clone integrations repo.
  • Install the elastic package locally.
  • Start the elastic stack using the elastic package.
  • Move to integrations/packages/ti_misp directory.
  • Run the following command to run tests.

elastic-package test -v

Related issues

@moxarth-rathod moxarth-rathod self-assigned this Jun 23, 2026
@moxarth-rathod
moxarth-rathod requested a review from a team as a code owner June 23, 2026 06:05
@moxarth-rathod moxarth-rathod added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jun 23, 2026
@moxarth-rathod
moxarth-rathod requested a review from a team as a code owner June 23, 2026 06:05
@moxarth-rathod moxarth-rathod added enhancement New feature or request Integration:ti_misp MISP Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels Jun 23, 2026
@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Jun 23, 2026

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@github-actions

github-actions Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

@vera-review-bot

vera-review-bot Bot commented Jun 23, 2026

Copy link
Copy Markdown

👀 I have started reviewing the PR

@vera-review-bot

vera-review-bot Bot commented Jun 23, 2026

Copy link
Copy Markdown

Vera Review Bot

For the current commit state, I did not find any issues.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@github-actions

github-actions Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

TL;DR

Buildkite failed in the ti_misp system tests because the httpjson input goes HEALTHY->DEGRADED when MISP pagination reaches an empty result page. This is triggered by the existing ti_misp pagination templates under the stack versions selected by the bumped Kibana constraint, not by the ingest pipeline edits.

Remediation

  • Do not rely on empty-template pagination termination for ti_misp on the ^8.19.2 || ^9.0.5 stack range; either adapt the pagination templates so termination does not emit failed to execute template page: the template result is empty, or defer the Kibana constraint bump until the httpjson do_not_log_failure status handling fix is available in the supported stack line.
  • Re-run elastic-package test system -v for packages/ti_misp and confirm the elastic-agent logs checks no longer report HEALTHY->DEGRADED for the default and daily-refetch cases.
Investigation details

Root Cause

The failing templates are in the httpjson stream definitions:

  • packages/ti_misp/data_stream/threat/agent/stream/httpjson.yml.hbs:75-81
  • packages/ti_misp/data_stream/threat_attributes/agent/stream/httpjson.yml.hbs:110-116

Both set body.page with templates that intentionally render an empty value when there are no more MISP results:

value: '[[if (ne (len .last_response.body.response) 0)]][[add .last_response.page 2]][[end]]'
fail_on_template_error: true
do_not_log_failure: true

and:

value: '[[if (ne (len .last_response.body.response.Attribute) 0)]][[add .last_response.page 2]][[end]]'
fail_on_template_error: true
do_not_log_failure: true

The PR also changes packages/ti_misp/manifest.yml to require ^8.19.2 || ^9.0.5. In those stack lines, the httpjson transform still reports the empty template to unit health before pagination treats it as the stop signal, so the system test log scan fails.

Evidence

system test: (elastic-agent logs - default) in ti_misp.threat
found error "Unit state changed httpjson-default-httpjson-ti_misp-... (HEALTHY->DEGRADED): failed to execute template page: the template result is empty"

system test: (elastic-agent logs - daily-refetch) in ti_misp.threat_attributes
found error "Unit state changed httpjson-default-httpjson-ti_misp-... (HEALTHY->DEGRADED): failed to execute template page: the template result is empty"

The PR diff does not change these httpjson stream templates, so the failure is exposed by the compatibility/version change rather than the remove_message ingest pipeline tag changes.

Verification

  • Reviewed the failed Buildkite annotation and PR diff.
  • Local system tests were not run because this environment does not support the Docker stack required by elastic-package test system.

Follow-up

If the package must keep ^8.19.2 || ^9.0.5 for agentless support, this likely needs either a package-level pagination workaround that avoids empty template errors or a stack-side backport of the httpjson do_not_log_failure status suppression for request pagination transforms.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

kcreddy
kcreddy reviewed Jun 23, 2026
View reviewed changes

@kcreddy kcreddy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@moxarth-rathod can you fix the system tests failure in CI?

@moxarth-rathod

moxarth-rathod commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

/test

@moxarth-rathod
moxarth-rathod enabled auto-merge (squash) June 26, 2026 03:33
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

✅ All changelog entries have the correct PR link.

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

Package ti_misp 👍(0) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
threat 2277.9 1574.8 -703.1 (-30.87%) 💔
threat_attributes 3731.34 2680.97 -1050.37 (-28.15%) 💔

To see the full report comment with /test benchmark fullreport

@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Jun 29, 2026

Copy link
Copy Markdown

💚 Build Succeeded

History

cc @moxarth-rathod

@moxarth-rathod
moxarth-rathod merged commit 1be02fb into elastic:main Jun 29, 2026
11 checks passed
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Package ti_misp - 1.44.0 containing this change is available at https://epr.elastic.co/package/ti_misp/1.44.0/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kcreddy kcreddy kcreddy approved these changes

Assignees

@moxarth-rathod moxarth-rathod

Labels

documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:ti_misp MISP Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[ti_misp] Add Agentless Deployment Support

2 participants

@moxarth-rathod @kcreddy