-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Description
Actual behavior
The synchronization fails as evidenced by the excerpts below. The event also becomes inaccessible in the browser and through the API
<CRONJOB:LOG FILE> [60951]: error -- Pulling an event (#ca48f90a-3a21-4b24-8092-e8f49119da3b) from Server #5 has failed. The sync process was not interrupted. -- SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '172.64.149.23' for key 'tmp_source_values.PRIMARY'
2025-09-26 15:06:32 Error: [PDOException] SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '172.64.149.23' for key 'tmp_source_values.PRIMARY'
Request URL: /events/view/2114323
User (User #232) @ 10.0.1.234
Error in: /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php, Line: 502
Stack Trace:
#0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(502): PDOStatement->execute()
#1 /var/www/MISP/app/Model/Datasource/Database/MysqlObserverExtended.php(71): DboSource->_execute()
#2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(715): MysqlObserverExtended->execute()
#3 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(648): DboSource->fetchAll()
#4 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(3490): DboSource->query()
#5 /var/www/MISP/app/Model/Behavior/OnDemandCorrelationBehavior.php(207): Model->query()
#6 /var/www/MISP/app/Model/Behavior/OnDemandCorrelationBehavior.php(528): OnDemandCorrelationBehavior->__collectCorrelations()
#7 /var/www/MISP/app/Model/Behavior/OnDemandCorrelationBehavior.php(515): OnDemandCorrelationBehavior->__filterRelatedEvents()
#8 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/BehaviorCollection.php(238): OnDemandCorrelationBehavior->fetchRelatedEventIds()
#9 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(830): BehaviorCollection->dispatchMethod()
#10 /var/www/MISP/app/Model/Correlation.php(1142): Model->__call()
#11 /var/www/MISP/app/Model/Event.php(745): Correlation->getRelatedEventIds()
#12 /var/www/MISP/app/Model/Event.php(2190): Event->getRelatedEvents()
#13 /var/www/MISP/app/Controller/EventsController.php(1828): Event->fetchEvent()
#14 [internal function]: EventsController->view()
#15 /var/www/MISP/app/Lib/cakephp/lib/Cake/Controller/Controller.php(500): ReflectionMethod->invokeArgs()
#16 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(193): Controller->invokeAction()
#17 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#18 /var/www/MISP/app/webroot/index.php(107): Dispatcher->dispatch()
(Pdb) x = misp.get_event(2114323)
2025-09-26T18:39:40Z CRITICAL Unknown error: the response is not in JSON.
Something is broken server-side, please send us everything that follows (careful with the auth key):
Request headers:
{'User-Agent': 'PyMISP 2.5.17 - Python 3.11', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'CAKEPHP=a7a36fbd256161c3987aefe1617297ff', 'content-type': 'application/json'}
Request body:
None
Response (if any):
{"name":"An Internal Error Has Occurred.","message":"An Internal Error Has Occurred.","url":"/events/view/2114323?nauth="}... (enable debug mode for more details)
*** pymisp.exceptions.MISPServerError: Error code 500:
{"name":"An Internal Error Has Occurred.","message":"An Internal Error Has Occurred.","url":"/events/view/2114323?nauth="}
Expected behavior
Synchronizations work without error when using the new On-Demand Correlations as they did with No ACL Correlations
Steps to reproduce
- Have On-Demand Correlations enabled as the engine
- Get/Create an event which has two attributes (Category: Network activity, type: domain | IP), where value1 for each of the two attributes is a different domain, but value2 is the same IP. Below is an example of the two attributes from the perspective of the database
*************************** 1. row *************************** id: 710088471 event_id: 2114323 object_id: 0 object_relation: NULL category: Network activity type: domain|ip value1: example1.domain.com value2: 172.64.149.23 to_ids: 1 uuid: 39b505ad-67e7-47f3-a35f-922fba677a2d timestamp: 1738709884 distribution: 5 sharing_group_id: 0 comment: deleted: 0 disable_correlation: 0 first_seen: 1704910545391318 last_seen: 1736889320023887 *************************** 2. row *************************** id: 710499302 event_id: 2114323 object_id: 0 object_relation: NULL category: Network activity type: domain|ip value1: example2.domain.com value2: 172.64.149.23 to_ids: 1 uuid: c47935b9-134c-4848-8d7d-796a2f74712c timestamp: 1704955627 distribution: 5 sharing_group_id: 0 comment: deleted: 0 disable_correlation: 0 first_seen: 1704946043000404 last_seen: 1704946043000404 - Run a synchronization on the event
Version
2.5.21
Operating System
Red Hat
Operating System version
8.10
PHP version
8.0.2
Browser
No response
Browser version
No response
Relevant log output
Here is the entire write-up and possible solution:
<CRONJOB:LOG FILE> [60951]: error -- Pulling an event (#ca48f90a-3a21-4b24-8092-e8f49119da3b) from Server #5 has failed. The sync process was not interrupted. -- SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '172.64.149.23' for key 'tmp_source_values.PRIMARY'
The UUID correlated to Event ID 2114323
When attempting access the event in the WebUI, the page showed "Internal Error Has Occurred". The corresponding MISP log file showed
2025-09-26 15:06:32 Error: [PDOException] SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '172.64.149.23' for key 'tmp_source_values.PRIMARY'
Request URL: /events/view/2114323
User <REDACTED> (User #232) @ 10.0.1.234
Error in: /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php, Line: 502
Stack Trace:
#0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(502): PDOStatement->execute()
#1 /var/www/MISP/app/Model/Datasource/Database/MysqlObserverExtended.php(71): DboSource->_execute()
#2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(715): MysqlObserverExtended->execute()
#3 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(648): DboSource->fetchAll()
#4 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(3490): DboSource->query()
#5 /var/www/MISP/app/Model/Behavior/OnDemandCorrelationBehavior.php(207): Model->query()
#6 /var/www/MISP/app/Model/Behavior/OnDemandCorrelationBehavior.php(528): OnDemandCorrelationBehavior->__collectCorrelations()
#7 /var/www/MISP/app/Model/Behavior/OnDemandCorrelationBehavior.php(515): OnDemandCorrelationBehavior->__filterRelatedEvents()
#8 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/BehaviorCollection.php(238): OnDemandCorrelationBehavior->fetchRelatedEventIds()
#9 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(830): BehaviorCollection->dispatchMethod()
#10 /var/www/MISP/app/Model/Correlation.php(1142): Model->__call()
#11 /var/www/MISP/app/Model/Event.php(745): Correlation->getRelatedEventIds()
#12 /var/www/MISP/app/Model/Event.php(2190): Event->getRelatedEvents()
#13 /var/www/MISP/app/Controller/EventsController.php(1828): Event->fetchEvent()
#14 [internal function]: EventsController->view()
#15 /var/www/MISP/app/Lib/cakephp/lib/Cake/Controller/Controller.php(500): ReflectionMethod->invokeArgs()
#16 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(193): Controller->invokeAction()
#17 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#18 /var/www/MISP/app/webroot/index.php(107): Dispatcher->dispatch()
I have attempted to access the event via API and had a similar result.
(Pdb) x = misp.get_event(2114323)
2025-09-26T18:39:40Z CRITICAL Unknown error: the response is not in JSON.
Something is broken server-side, please send us everything that follows (careful with the auth key):
Request headers:
{'User-Agent': 'PyMISP 2.5.17 - Python 3.11', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'CAKEPHP=a7a36fbd256161c3987aefe1617297ff', 'content-type': 'application/json'}
Request body:
None
Response (if any):
{"name":"An Internal Error Has Occurred.","message":"An Internal Error Has Occurred.","url":"\/events\/view\/2114323?nauth=<REDACTED>"}... (enable debug mode for more details)
*** pymisp.exceptions.MISPServerError: Error code 500:
{"name":"An Internal Error Has Occurred.","message":"An Internal Error Has Occurred.","url":"\/events\/view\/2114323?nauth=<REDACTED>"}
I proceeded to look at the MYSQL database, only to see that tmp_source_values is not an active table. The table is only references in: /var/www/MISP/app/Model/Behavior/OnDemandCorrelationBehavior.php starting on line 203.
203 CREATE TEMPORARY TABLE tmp_source_values (
204 value2 VARCHAR(64) PRIMARY KEY,
205 id INT(10) UNSIGNED NOT NULL
206 ) ENGINE=MEMORY;");
207 $this->Correlation->query("
208 INSERT INTO tmp_source_values (value2, id)
209 SELECT a.value2, a.id
210 FROM attributes a
211 WHERE
212 a.event_id = ? AND
213 a.value2 <> '' AND
214 a.deleted = 0 AND
215 a.disable_correlation = 0 AND
216 a.type IN ('" . implode("','", $this->value2CorrelatingTypes) . "')
217 ",
218 [$eventId]
219 );
220 $use_index = $this->customIndeces['attributes']['idx_val2_target'] ? 'FORCE INDEX (`idx_val2 _target`)' : '';
221 $sql = "
222 SELECT
223 target.id AS id,
224 target.event_id AS event_id,
225 target.value2 AS 'value',
226 target.type,
227 source.id AS source_id
228 FROM attributes AS target " . $use_index . "
229 JOIN tmp_source_values source
230 ON target.value2 = source.value2
231 WHERE
232 target.event_id != ? AND
233 target.value2 <> '' AND
234 target.deleted = 0 AND
235 target.disable_correlation = 0 AND
236 target.type IN ('" . implode("','", $this->value2CorrelatingTypes) . "')
237 LIMIT " . $max_correlations
238 ;
What is happening in the code is that a temporary table is being created using value2 as a primary key. A problem can arise if an event has multiple attributes with the same value2 which our event has
mysql> select count(*) from attributes where value2='172.64.149.23' and event_id=2114323;
+----------+
| count(*) |
+----------+
| 2 |
+----------+
mysql> select * from attributes where value2='172.64.149.23' and event_id=2114323 \G;
*************************** 1. row ***************************
id: 710088471
event_id: 2114323
object_id: 0
object_relation: NULL
category: Network activity
type: domain|ip
value1: <UNIQUE DOMAIN 1>
value2: 172.64.149.23
to_ids: 1
uuid: 39b505ad-67e7-47f3-a35f-922fba677a2d
timestamp: 1738709884
distribution: 5
sharing_group_id: 0
comment:
deleted: 0
disable_correlation: 0
first_seen: 1704910545391318
last_seen: 1736889320023887
*************************** 2. row ***************************
id: 710499302
event_id: 2114323
object_id: 0
object_relation: NULL
category: Network activity
type: domain|ip
value1: <UNIQUE DOMAIN 2>
value2: 172.64.149.23
to_ids: 1
uuid: c47935b9-134c-4848-8d7d-796a2f74712c
timestamp: 1704955627
distribution: 5
sharing_group_id: 0
comment:
deleted: 0
disable_correlation: 0
first_seen: 1704946043000404
last_seen: 1704946043000404
ERROR:
No query specified
We can see in the above example that value1 differs but value2 is the same.
The solution is to make the temporary table creation be a combination of value1 and value2. The second point is to prevent correlation construction on event syncs. Correlations are necessary for users, or API pulls, but event syncs do not display data in such a way and should avoid doing so.Extra attachments
No response
Code of Conduct
- I agree to follow this project's Code of Conduct