Skip to content

Bug: Duplicate Values Break Sync When Using On-Demand Correlations #10521

@emv000

Description

@emv000

Actual behavior

The synchronization fails as evidenced by the excerpts below. The event also becomes inaccessible in the browser and through the API

<CRONJOB:LOG FILE> [60951]: error -- Pulling an event (#ca48f90a-3a21-4b24-8092-e8f49119da3b) from Server #5 has failed. The sync process was not interrupted. -- SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '172.64.149.23' for key 'tmp_source_values.PRIMARY'

2025-09-26 15:06:32 Error: [PDOException] SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '172.64.149.23' for key 'tmp_source_values.PRIMARY'
Request URL: /events/view/2114323
User (User #232) @ 10.0.1.234
Error in: /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php, Line: 502
Stack Trace:
#0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(502): PDOStatement->execute()
#1 /var/www/MISP/app/Model/Datasource/Database/MysqlObserverExtended.php(71): DboSource->_execute()
#2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(715): MysqlObserverExtended->execute()
#3 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(648): DboSource->fetchAll()
#4 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(3490): DboSource->query()
#5 /var/www/MISP/app/Model/Behavior/OnDemandCorrelationBehavior.php(207): Model->query()
#6 /var/www/MISP/app/Model/Behavior/OnDemandCorrelationBehavior.php(528): OnDemandCorrelationBehavior->__collectCorrelations()
#7 /var/www/MISP/app/Model/Behavior/OnDemandCorrelationBehavior.php(515): OnDemandCorrelationBehavior->__filterRelatedEvents()
#8 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/BehaviorCollection.php(238): OnDemandCorrelationBehavior->fetchRelatedEventIds()
#9 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(830): BehaviorCollection->dispatchMethod()
#10 /var/www/MISP/app/Model/Correlation.php(1142): Model->__call()
#11 /var/www/MISP/app/Model/Event.php(745): Correlation->getRelatedEventIds()
#12 /var/www/MISP/app/Model/Event.php(2190): Event->getRelatedEvents()
#13 /var/www/MISP/app/Controller/EventsController.php(1828): Event->fetchEvent()
#14 [internal function]: EventsController->view()
#15 /var/www/MISP/app/Lib/cakephp/lib/Cake/Controller/Controller.php(500): ReflectionMethod->invokeArgs()
#16 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(193): Controller->invokeAction()
#17 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#18 /var/www/MISP/app/webroot/index.php(107): Dispatcher->dispatch()

(Pdb) x = misp.get_event(2114323)
2025-09-26T18:39:40Z CRITICAL Unknown error: the response is not in JSON.
Something is broken server-side, please send us everything that follows (careful with the auth key):
Request headers:
{'User-Agent': 'PyMISP 2.5.17 - Python 3.11', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'CAKEPHP=a7a36fbd256161c3987aefe1617297ff', 'content-type': 'application/json'}
Request body:
None
Response (if any):
{"name":"An Internal Error Has Occurred.","message":"An Internal Error Has Occurred.","url":"/events/view/2114323?nauth="}... (enable debug mode for more details)
*** pymisp.exceptions.MISPServerError: Error code 500:
{"name":"An Internal Error Has Occurred.","message":"An Internal Error Has Occurred.","url":"/events/view/2114323?nauth="}

Expected behavior

Synchronizations work without error when using the new On-Demand Correlations as they did with No ACL Correlations

Steps to reproduce

  1. Have On-Demand Correlations enabled as the engine
  2. Get/Create an event which has two attributes (Category: Network activity, type: domain | IP), where value1 for each of the two attributes is a different domain, but value2 is the same IP. Below is an example of the two attributes from the perspective of the database*************************** 1. row *************************** id: 710088471 event_id: 2114323 object_id: 0 object_relation: NULL category: Network activity type: domain|ip value1: example1.domain.com value2: 172.64.149.23 to_ids: 1 uuid: 39b505ad-67e7-47f3-a35f-922fba677a2d timestamp: 1738709884 distribution: 5 sharing_group_id: 0 comment: deleted: 0 disable_correlation: 0 first_seen: 1704910545391318 last_seen: 1736889320023887 *************************** 2. row *************************** id: 710499302 event_id: 2114323 object_id: 0 object_relation: NULL category: Network activity type: domain|ip value1: example2.domain.com value2: 172.64.149.23 to_ids: 1 uuid: c47935b9-134c-4848-8d7d-796a2f74712c timestamp: 1704955627 distribution: 5 sharing_group_id: 0 comment: deleted: 0 disable_correlation: 0 first_seen: 1704946043000404 last_seen: 1704946043000404
  3. Run a synchronization on the event

Version

2.5.21

Operating System

Red Hat

Operating System version

8.10

PHP version

8.0.2

Browser

No response

Browser version

No response

Relevant log output

Here is the entire write-up and possible solution:

<CRONJOB:LOG FILE> [60951]: error -- Pulling an event (#ca48f90a-3a21-4b24-8092-e8f49119da3b) from Server #5 has failed. The sync process was not interrupted. -- SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '172.64.149.23' for key 'tmp_source_values.PRIMARY'

The UUID correlated to Event ID 2114323

When attempting access the event in the WebUI, the page showed "Internal Error Has Occurred". The corresponding MISP log file showed

2025-09-26 15:06:32 Error: [PDOException] SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '172.64.149.23' for key 'tmp_source_values.PRIMARY'
Request URL: /events/view/2114323
User <REDACTED> (User #232) @ 10.0.1.234
Error in: /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php, Line: 502
Stack Trace:
#0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(502): PDOStatement->execute()
#1 /var/www/MISP/app/Model/Datasource/Database/MysqlObserverExtended.php(71): DboSource->_execute()
#2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(715): MysqlObserverExtended->execute()
#3 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(648): DboSource->fetchAll()
#4 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(3490): DboSource->query()
#5 /var/www/MISP/app/Model/Behavior/OnDemandCorrelationBehavior.php(207): Model->query()
#6 /var/www/MISP/app/Model/Behavior/OnDemandCorrelationBehavior.php(528): OnDemandCorrelationBehavior->__collectCorrelations()
#7 /var/www/MISP/app/Model/Behavior/OnDemandCorrelationBehavior.php(515): OnDemandCorrelationBehavior->__filterRelatedEvents()
#8 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/BehaviorCollection.php(238): OnDemandCorrelationBehavior->fetchRelatedEventIds()
#9 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(830): BehaviorCollection->dispatchMethod()
#10 /var/www/MISP/app/Model/Correlation.php(1142): Model->__call()
#11 /var/www/MISP/app/Model/Event.php(745): Correlation->getRelatedEventIds()
#12 /var/www/MISP/app/Model/Event.php(2190): Event->getRelatedEvents()
#13 /var/www/MISP/app/Controller/EventsController.php(1828): Event->fetchEvent()
#14 [internal function]: EventsController->view()
#15 /var/www/MISP/app/Lib/cakephp/lib/Cake/Controller/Controller.php(500): ReflectionMethod->invokeArgs()
#16 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(193): Controller->invokeAction()
#17 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#18 /var/www/MISP/app/webroot/index.php(107): Dispatcher->dispatch()

I have attempted to access the event via API and had a similar result.

(Pdb) x = misp.get_event(2114323)
2025-09-26T18:39:40Z CRITICAL Unknown error: the response is not in JSON.
Something is broken server-side, please send us everything that follows (careful with the auth key):
Request headers:
{'User-Agent': 'PyMISP 2.5.17 - Python 3.11', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'CAKEPHP=a7a36fbd256161c3987aefe1617297ff', 'content-type': 'application/json'}
Request body:
None
Response (if any):
{"name":"An Internal Error Has Occurred.","message":"An Internal Error Has Occurred.","url":"\/events\/view\/2114323?nauth=<REDACTED>"}... (enable debug mode for more details)
*** pymisp.exceptions.MISPServerError: Error code 500:
{"name":"An Internal Error Has Occurred.","message":"An Internal Error Has Occurred.","url":"\/events\/view\/2114323?nauth=<REDACTED>"}

I proceeded to look at the MYSQL database, only to see that tmp_source_values is not an active table. The table is only references in: /var/www/MISP/app/Model/Behavior/OnDemandCorrelationBehavior.php starting on line 203.

    203                 CREATE TEMPORARY TABLE tmp_source_values (
    204                     value2 VARCHAR(64) PRIMARY KEY,
    205                     id INT(10) UNSIGNED NOT NULL
    206                 ) ENGINE=MEMORY;");
    207                 $this->Correlation->query("
    208                         INSERT INTO tmp_source_values (value2, id)
    209                         SELECT a.value2, a.id
    210                         FROM attributes a
    211                         WHERE
    212                             a.event_id = ? AND
    213                             a.value2 <> '' AND
    214                             a.deleted = 0 AND
    215                             a.disable_correlation = 0 AND
    216                             a.type IN ('" . implode("','", $this->value2CorrelatingTypes) . "')
    217                     ",
    218                     [$eventId]
    219                 );
    220                 $use_index = $this->customIndeces['attributes']['idx_val2_target'] ? 'FORCE INDEX (`idx_val2        _target`)' : '';
    221                 $sql = "
    222                     SELECT
    223                         target.id AS id,
    224                         target.event_id AS event_id,
    225                         target.value2 AS 'value',
    226                         target.type,
    227                         source.id AS source_id
    228                     FROM attributes AS target " . $use_index . "
    229                     JOIN tmp_source_values source
    230                     ON target.value2 = source.value2
    231                     WHERE
    232                         target.event_id != ? AND
    233                         target.value2 <> '' AND
    234                         target.deleted = 0 AND
    235                         target.disable_correlation = 0 AND
    236                         target.type IN ('" . implode("','", $this->value2CorrelatingTypes) . "')
    237                     LIMIT " . $max_correlations
    238                 ;

What is happening in the code is that a temporary table is being created using value2 as a primary key. A problem can arise if an event has multiple attributes with the same value2 which our event has

mysql> select count(*) from attributes where value2='172.64.149.23' and event_id=2114323;
+----------+
| count(*) |
+----------+
|        2 |
+----------+
mysql> select * from attributes where value2='172.64.149.23' and event_id=2114323 \G;
*************************** 1. row ***************************
                 id: 710088471
           event_id: 2114323
          object_id: 0
    object_relation: NULL
           category: Network activity
               type: domain|ip
             value1: <UNIQUE DOMAIN 1>
             value2: 172.64.149.23
             to_ids: 1
               uuid: 39b505ad-67e7-47f3-a35f-922fba677a2d
          timestamp: 1738709884
       distribution: 5
   sharing_group_id: 0
            comment:
            deleted: 0
disable_correlation: 0
         first_seen: 1704910545391318
          last_seen: 1736889320023887
*************************** 2. row ***************************
                 id: 710499302
           event_id: 2114323
          object_id: 0
    object_relation: NULL
           category: Network activity
               type: domain|ip
             value1: <UNIQUE DOMAIN 2>
             value2: 172.64.149.23
             to_ids: 1
               uuid: c47935b9-134c-4848-8d7d-796a2f74712c
          timestamp: 1704955627
       distribution: 5
   sharing_group_id: 0
            comment:
            deleted: 0
disable_correlation: 0
         first_seen: 1704946043000404
          last_seen: 1704946043000404
ERROR:
No query specified
We can see in the above example that value1 differs but value2 is the same.

The solution is to make the temporary table creation be a combination of value1 and value2. The second point is to prevent correlation construction on event syncs. Correlations are necessary for users, or API pulls, but event syncs do not display data in such a way and should avoid doing so.

Extra attachments

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs triageThis issue has been automatically labelled and needs further triage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions