Skip to content

[SIEM] Removing a Rule's Action breaks Rule execution #64870

Description

@tsg

[reported originally by @rylnd)

Kibana version: 7.7.0

After an action is removed from a rule, the rule fails to execute. To resolve this, disable and then re-enable the rule.

This is present on master and 7.7.

Steps to Reproduce

  1. Create and activate a Rule with an Action
    • any action type and frequency work, but easiest to test with on each rule execution
    • similar to above, running the rule every 30s eases testing
  2. (optional): wait for the rule to execute, generate signals, and fire the action
    • this is the pre-bug sanity check
  3. Edit the Rule and "deactivate" its Action by changing its Actions Frequency to perform no actions
  4. Observe errant behaviors:
    • Rule no longer generates signals
    • Rule no longer reports success/failure (Last response will remain as it was prior to step 3)
    • if running locally, kibana logs should show an error:
      server    log   [12:20:57.406] [error][encryptedSavedObjects][plugins] Failed to decrypt "apiKey" attribute: Unsupported state or unable to authenticate data
      server    log   [12:20:57.406] [error][alerting][alerting][plugins][plugins] Executing Alert "684c28a5-a49d-490b-9c9b-26dc865e641c" has resulted in Error: Unable to decrypt attribute "apiKey"
      

NB That querying the corresponding task, we find that it still exists in the "idle" state:

GET .kibana_task_manager*/_search
{"query":{"match":{"task.taskType":"alerting:siem.signals"}}}

Activity

  1. added
    bugFixes for quality problems that affect the customer experience
    on Apr 30, 2020
  2. elasticmachine commented on Apr 30, 2020

    @elasticmachine
    Contributor

    Pinging @elastic/siem (Team:SIEM)

  3. changed the title [-]Removing a Rule's Action breaks Rule execution[/-] [+][SIEM] Removing a Rule's Action breaks Rule execution[/+] on Apr 30, 2020
  4. rylnd commented on May 8, 2020

    @rylnd
    Contributor

    This is a consequence of #53868, which is currently blocked and under discussion.

  5. rylnd commented on May 13, 2020

    @rylnd
    Contributor

    Dev update: we have a temporary solution in the following diff:

    diff --git a/x-pack/plugins/siem/server/lib/detection_engine/routes/rules/update_rules_route.ts b/x-pack/plugins/siem/server/lib/detection_engine/routes/rules/update_rules_route.ts
    index f15154a096..29dd52cc57 100644
    --- a/x-pack/plugins/siem/server/lib/detection_engine/routes/rules/update_rules_route.ts
    +++ b/x-pack/plugins/siem/server/lib/detection_engine/routes/rules/update_rules_route.ts
    @@ -138,6 +138,7 @@ export const updateRulesRoute = (router: IRouter, ml: SetupPlugins['ml']) => {
                 ruleActions,
                 ruleStatuses.saved_objects[0]
               );
    +          await alertsClient.updateApiKey({ id: rule.id });
               if (errors != null) {
                 return siemResponse.error({ statusCode: 500, body: errors });
               } else {

    It's not ideal as the call to updateApiKey itself logs an error when removing the rule's action, but it does allow the rule to continue executing sans actions. If acceptable we'll have to add a similar fix to all patch/update rule routes that can delete actions.

    However, we're still digging into the underlying cause here, so we'll be roping in Alerting for some assistance before this ships.

  6. mikecote commented on May 26, 2020

    @mikecote
    Contributor

    Failed to decrypt "apiKey" attribute: Unsupported state or unable to authenticate data usually means data has change since it was last encrypted. This can happen when doing a partial update or when the Elasticsearch document merges json on update. You can tell the latter by comparing alert params on update vs what is stored in the document in ES after update.

    I don't think #53868 will solve the issue because that is to handle invalidating API keys where this issue is trying to decrypt the API key from a saved object.

  7. mikecote commented on May 26, 2020

    @mikecote
    Contributor

    If it's confirmed that the alert params get merged with old values on update, this file x-pack/plugins/siem/server/lib/detection_engine/signals/signal_params_schema.ts would need to set default null values to all the optional properties.

  8. rylnd commented on May 27, 2020

    @rylnd
    Contributor

    7.7: #67426

  9. MadameSheema commented on May 29, 2020

    @MadameSheema
    Contributor

    Fixed in 7.7.1 :)

  10. added 6 commits that reference this issue on Jun 3, 2020
    508f860
    5fd6416
    6e1334c
    f70447f
    7425b39
    fdeed82
  11. added
    Team: SecuritySolutionSecurity Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
    on Oct 27, 2020
  12. added a commit that references this issue on May 6, 2026
    9d57037
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Team: SecuritySolutionSecurity Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.Team:SIEMbugFixes for quality problems that affect the customer experience

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions