Skip to content

ThreadSanitizer SEGV Fix: Race Condition#33593

Merged
juliancnn merged 1 commit into
mainfrom
bug/32387-Analyze_TSAN
Dec 31, 2025
Merged

ThreadSanitizer SEGV Fix: Race Condition#33593
juliancnn merged 1 commit into
mainfrom
bug/32387-Analyze_TSAN

Conversation

@jam300
Copy link
Copy Markdown
Member

@jam300 jam300 commented Dec 19, 2025 •
edited
Loading

Description

Spike #32387 ThreadSanitizer detected multiple critical race conditions in the router orchestrator and logging subsystems that caused segmentation faults and test failures.

Issue 1: Router Orchestrator Race Condition

  • Error Type: SEGV (Segmentation Fault)
  • Memory Address: 0x000000000000 (null pointer dereference)
  • Location: orchestrator.cpp:449 in getEntries()
  • Root Cause: Time-Of-Check-To-Time-Of-Use (TOCTOU) vulnerability
Tsan_report 

==256415==ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000000 (pc 0x559009e7a933 bp 0x7fab2c3e6af0 sp 0x7fab2c3e6aa0 T256727)
==256415==The signal is caused by a READ memory access.
==256415==Hint: address points to the zero page.
    #0 router::Orchestrator::getEntries[abi:cxx11]() const /workspaces/Wazuh_5x/wazuh/src/engine/source/router/src/orchestrator.cpp:449 (wazuh-analysisdTsan+0x124a933) (BuildId: dfeb9fcefa84a8cd6cdf7de0e555041b5c1a47e6)
    #1 operator() /workspaces/Wazuh_5x/wazuh/src/engine/source/main.cpp:584 (wazuh-analysisdTsan+0xd73338) (BuildId: dfeb9fcefa84a8cd6cdf7de0e555041b5c1a47e6)
    #2 __invoke_impl<void, main(int, char**)::<lambda()>&> /usr/include/c++/14/bits/invoke.h:61 (wazuh-analysisdTsan+0xd7e857) (BuildId: dfeb9fcefa84a8cd6cdf7de0e555041b5c1a47e6)
    #3 __invoke_r<void, main(int, char**)::<lambda()>&> /usr/include/c++/14/bits/invoke.h:111 (wazuh-analysisdTsan+0xd7d86c) (BuildId: dfeb9fcefa84a8cd6cdf7de0e555041b5c1a47e6)
    #4 _M_invoke /usr/include/c++/14/bits/std_function.h:290 (wazuh-analysisdTsan+0xd7c2be) (BuildId: dfeb9fcefa84a8cd6cdf7de0e555041b5c1a47e6)
    #5 std::function<void ()>::operator()() const /usr/include/c++/14/bits/std_function.h:591 (wazuh-analysisdTsan+0xdbfbde) (BuildId: dfeb9fcefa84a8cd6cdf7de0e555041b5c1a47e6)
    #6 scheduler::Scheduler::executeTask(scheduler::TaskQueue::TaskItem const&) /workspaces/Wazuh_5x/wazuh/src/engine/source/scheduler/src/scheduler.cpp:217 (wazuh-analysisdTsan+0x12f39f4) (BuildId: dfeb9fcefa84a8cd6cdf7de0e555041b5c1a47e6)
    #7 scheduler::Scheduler::workerThread() /workspaces/Wazuh_5x/wazuh/src/engine/source/scheduler/src/scheduler.cpp:177 (wazuh-analysisdTsan+0x12f351f) (BuildId: dfeb9fcefa84a8cd6cdf7de0e555041b5c1a47e6)
    #8 void std::__invoke_impl<void, void (scheduler::Scheduler::*)(), scheduler::Scheduler*>(std::__invoke_memfun_deref, void (scheduler::Scheduler::*&&)(), scheduler::Scheduler*&&) /usr/include/c++/14/bits/invoke.h:74 (wazuh-analysisdTsan+0x12fc091) (BuildId: dfeb9fcefa84a8cd6cdf7de0e555041b5c1a47e6)
    #9 std::__invoke_result<void (scheduler::Scheduler::*)(), scheduler::Scheduler*>::type std::__invoke<void (scheduler::Scheduler::*)(), scheduler::Scheduler*>(void (scheduler::Scheduler::*&&)(), scheduler::Scheduler*&&) /usr/include/c++/14/bits/invoke.h:96 (wazuh-analysisdTsan+0x12fbf5e) (BuildId: dfeb9fcefa84a8cd6cdf7de0e555041b5c1a47e6)
    #10 void std::thread::_Invoker<std::tuple<void (scheduler::Scheduler::*)(), scheduler::Scheduler*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/include/c++/14/bits/std_thread.h:301 (wazuh-analysisdTsan+0x12fbe5a) (BuildId: dfeb9fcefa84a8cd6cdf7de0e555041b5c1a47e6)
    #11 std::thread::_Invoker<std::tuple<void (scheduler::Scheduler::*)(), scheduler::Scheduler*> >::operator()() /usr/include/c++/14/bits/std_thread.h:308 (wazuh-analysisdTsan+0x12fbd72) (BuildId: dfeb9fcefa84a8cd6cdf7de0e555041b5c1a47e6)
    #12 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (scheduler::Scheduler::*)(), scheduler::Scheduler*> > >::_M_run() /usr/include/c++/14/bits/std_thread.h:253 (wazuh-analysisdTsan+0x12fbce8) (BuildId: dfeb9fcefa84a8cd6cdf7de0e555041b5c1a47e6)
    #13 execute_native_thread_routine <null> (wazuh-analysisdTsan+0x1944393) (BuildId: dfeb9fcefa84a8cd6cdf7de0e555041b5c1a47e6)
    #14 __tsan_thread_start_func ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1012 (libtsan.so.2+0x4edee) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
    #15 start_thread nptl/pthread_create.c:447 (libc.so.6+0x9caa3) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)
    #16 clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 (libc.so.6+0x129c6b) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)

ThreadSanitizer can not provide additional info.
SUMMARY: ThreadSanitizer: SEGV /workspaces/Wazuh_5x/wazuh/src/engine/source/router/src/orchestrator.cpp:449 in router::Orchestrator::getEntries[abi:cxx11]() const
==256415==ABORTING

Race Condition Flow:

  1. Scheduler thread calls getEntries() and acquires shared_lock
  2. Cleanup thread calls cleanup() with incorrect shared_lock
  3. Cleanup clears m_routerWorkers while other threads can still read it
  4. Scheduler thread calls front() on empty list

Root Cause: Race condition between scheduler threads calling getEntries() and cleanup operations. The cleanup() method used a shared_lock (read lock) instead of unique_lock (write lock), allowing concurrent reads while clearing m_routerWorkers, resulting in null pointer dereference.

Issue 2: Router Orchestrator Deadlock

Problem: After fixing the initial race condition, integration tests revealed a deadlock caused by attempting to acquire locks twice in nested function calls.

TSAN Report Summary:

  • Error Type: Deadlock / Lock ordering issue
  • Location: orchestrator.cpp - multiple dump*() methods
  • Root Cause: Lock re-acquisition in nested calls

Deadlock Flow:

  1. Public method acquires unique_lock (e.g., postEntry(), deleteEntry())
  2. Same method calls dumpRouters() or dumpTesters()
  3. dump*() methods attempt to acquire shared_lock again
  4. Deadlock: Thread already holds a lock and tries to acquire another
base::OptError Orchestrator::postEntry(const prod::EntryPost& entry) {
    std::unique_lock lock {m_syncMutex};  // ← Lock acquired
    // ... operations ...
    dumpRouters();  // ← Tries to acquire shared_lock AGAIN → DEADLOCK
}

Solution: Introduced *Internal variants of dump methods that assume lock is already held:
dumpTestersInternal() - no lock acquisition
dumpRoutersInternal() - no lock acquisition
dumpEpsInternal() - no lock acquisition
Public methods (dumpTesters(), dumpRouters(), dumpEps()) acquire locks and call Internal variants.

Issue 3: Logging Subsystem Data Races

Problem: ChannelHandler tests failed due to data races in CustomSink.

TSAN Report Summary:

  • Error Type: Data race (concurrent unsynchronized writes)
  • Memory Address: 0x720c0000004c (shared mutable state in CustomSink)
  • Locations:
    logging.cpp:37 in CustomSink::log()
    logging.cpp:55 in CustomSink::flush()
    spdlog/pattern_formatter-inl.h:970
  • Root Cause: CustomSink not thread-safe despite multi-threaded usage
TSAN Report - CustomSink Data Race 

==================
6882: WARNING: ThreadSanitizer: data race (pid=114085)
6882:   Write of size 4 at 0x720c0000004c by thread T5:
6882:     #0 logging::CustomSink::log(spdlog::details::log_msg const&) /workspaces/Wazuh_5x/wazuh/src/engine/source/base/src/logging.cpp:37 (streamlogger_utest+0x68a664) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #1 spdlog::logger::sink_it_(spdlog::details::log_msg const&) /workspaces/Wazuh_5x/wazuh/src/external/spdlog/include/spdlog/logger-inl.h:138 (streamlogger_utest+0x3c02cf) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #2 spdlog::logger::log_it_(spdlog::details::log_msg const&, bool, bool) /workspaces/Wazuh_5x/wazuh/src/external/spdlog/include/spdlog/logger-inl.h:128 (streamlogger_utest+0x56632e) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #3 void spdlog::logger::log_<char const*&, int>(spdlog::source_loc, spdlog::level::level_enum, fmt::v8::basic_string_view<char>, char const*&, int&&) /workspaces/Wazuh_5x/wazuh/src/external/spdlog/include/spdlog/logger.h:332 (streamlogger_utest+0x568449) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #4 void spdlog::logger::log<char const*&, int>(spdlog::source_loc, spdlog::level::level_enum, fmt::v8::basic_format_string<char, fmt::v8::type_identity<char const*&>::type, fmt::v8::type_identity<int>::type>, char const*&, int&&) /workspaces/Wazuh_5x/wazuh/src/external/spdlog/include/spdlog/logger.h:80 (streamlogger_utest+0x567626) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #5 logging::backend_log(logging::Level, char const*, int, char const*, char const*, unsigned long) /workspaces/Wazuh_5x/wazuh/src/engine/source/base/include/base/logging.hpp:274 (streamlogger_utest+0x566cf5) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #6 void logging::log_bridge<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(logging::Level, char const*, int, char const*, fmt::v8::basic_format_string<char, fmt::v8::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /workspaces/Wazuh_5x/wazuh/src/engine/source/base/include/base/logging.hpp:328 (streamlogger_utest+0x567b30) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #7 streamlog::ChannelHandler::workerThreadFunc() /workspaces/Wazuh_5x/wazuh/src/engine/source/streamlog/src/channel.cpp:282 (streamlogger_utest+0x570e26) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #8 void std::__invoke_impl<void, void (streamlog::ChannelHandler::*)(), streamlog::ChannelHandler*>(std::__invoke_memfun_deref, void (streamlog::ChannelHandler::*&&)(), streamlog::ChannelHandler*&&) /usr/include/c++/14/bits/invoke.h:74 (streamlogger_utest+0x5d7d4e) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #9 std::__invoke_result<void (streamlog::ChannelHandler::*)(), streamlog::ChannelHandler*>::type std::__invoke<void (streamlog::ChannelHandler::*)(), streamlog::ChannelHandler*>(void (streamlog::ChannelHandler::*&&)(), streamlog::ChannelHandler*&&) /usr/include/c++/14/bits/invoke.h:96 (streamlogger_utest+0x5d7b8d) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #10 void std::thread::_Invoker<std::tuple<void (streamlog::ChannelHandler::*)(), streamlog::ChannelHandler*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/include/c++/14/bits/std_thread.h:301 (streamlogger_utest+0x5d7a2c) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #11 std::thread::_Invoker<std::tuple<void (streamlog::ChannelHandler::*)(), streamlog::ChannelHandler*> >::operator()() /usr/include/c++/14/bits/std_thread.h:308 (streamlogger_utest+0x5d77b0) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #12 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (streamlog::ChannelHandler::*)(), streamlog::ChannelHandler*> > >::_M_run() /usr/include/c++/14/bits/std_thread.h:253 (streamlogger_utest+0x5d74ac) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #13 <null> <null> (libstdc++.so.6+0xecdb3) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)
6882: 
6882:   Previous write of size 4 at 0x720c0000004c by thread T4 (mutexes: write M0):
6882:     #0 logging::CustomSink::log(spdlog::details::log_msg const&) /workspaces/Wazuh_5x/wazuh/src/engine/source/base/src/logging.cpp:37 (streamlogger_utest+0x68a664) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #1 spdlog::logger::sink_it_(spdlog::details::log_msg const&) /workspaces/Wazuh_5x/wazuh/src/external/spdlog/include/spdlog/logger-inl.h:138 (streamlogger_utest+0x3c02cf) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #2 spdlog::logger::log_it_(spdlog::details::log_msg const&, bool, bool) /workspaces/Wazuh_5x/wazuh/src/external/spdlog/include/spdlog/logger-inl.h:128 (streamlogger_utest+0x56632e) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #3 void spdlog::logger::log_<char const*&, int>(spdlog::source_loc, spdlog::level::level_enum, fmt::v8::basic_string_view<char>, char const*&, int&&) /workspaces/Wazuh_5x/wazuh/src/external/spdlog/include/spdlog/logger.h:332 (streamlogger_utest+0x568449) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #4 void spdlog::logger::log<char const*&, int>(spdlog::source_loc, spdlog::level::level_enum, fmt::v8::basic_format_string<char, fmt::v8::type_identity<char const*&>::type, fmt::v8::type_identity<int>::type>, char const*&, int&&) /workspaces/Wazuh_5x/wazuh/src/external/spdlog/include/spdlog/logger.h:80 (streamlogger_utest+0x567626) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #5 logging::backend_log(logging::Level, char const*, int, char const*, char const*, unsigned long) /workspaces/Wazuh_5x/wazuh/src/engine/source/base/include/base/logging.hpp:274 (streamlogger_utest+0x566cf5) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #6 void logging::log_bridge<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(logging::Level, char const*, int, char const*, fmt::v8::basic_format_string<char, fmt::v8::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /workspaces/Wazuh_5x/wazuh/src/engine/source/base/include/base/logging.hpp:328 (streamlogger_utest+0x567b30) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #7 streamlog::ChannelHandler::createWriter() /workspaces/Wazuh_5x/wazuh/src/engine/source/streamlog/src/channel.cpp:693 (streamlogger_utest+0x5744c8) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #8 operator() /workspaces/Wazuh_5x/wazuh/src/engine/source/streamlog/test/src/unit/channel_test.cpp:957 (streamlogger_utest+0x4bed51) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #9 __invoke_impl<void, ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > /usr/include/c++/14/bits/invoke.h:61 (streamlogger_utest+0x4db71f) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #10 __invoke<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > /usr/include/c++/14/bits/invoke.h:96 (streamlogger_utest+0x4db56e) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #11 _M_invoke<0> /usr/include/c++/14/bits/std_thread.h:301 (streamlogger_utest+0x4db3d6) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #12 operator() /usr/include/c++/14/bits/std_thread.h:308 (streamlogger_utest+0x4db22c) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #13 operator() /usr/include/c++/14/future:1439 (streamlogger_utest+0x4dac27) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #14 __invoke_impl<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>&> /usr/include/c++/14/bits/invoke.h:61 (streamlogger_utest+0x4da5fa) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #15 __invoke_r<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>&> /usr/include/c++/14/bits/invoke.h:114 (streamlogger_utest+0x4d9e49) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #16 _M_invoke /usr/include/c++/14/bits/std_function.h:291 (streamlogger_utest+0x4d95b2) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #17 std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const /usr/include/c++/14/bits/std_function.h:591 (streamlogger_utest+0x4ea6fe) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #18 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) /usr/include/c++/14/future:596 (streamlogger_utest+0x4e40b4) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #19 void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/14/bits/invoke.h:74 (streamlogger_utest+0x4fb9a6) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #20 std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/14/bits/invoke.h:96 (streamlogger_utest+0x4f3a80) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #21 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const /usr/include/c++/14/mutex:909 (streamlogger_utest+0x4ea507) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #22 std::once_flag::_Prepare_execution::_Prepare_execution<std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}>(void (std::__future_base::_State_baseV2::*&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*))::{lambda()#1}::operator()() const /usr/include/c++/14/mutex:845 (streamlogger_utest+0x4f3af6) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #23 std::once_flag::_Prepare_execution::_Prepare_execution<std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}>(void (std::__future_base::_State_baseV2::*&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*))::{lambda()#1}::_FUN() /usr/include/c++/14/mutex:845 (streamlogger_utest+0x4f3b3d) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #24 pthread_once ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1587 (libtsan.so.2+0x57ff5) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
6882:     #25 __gthread_once /usr/include/x86_64-linux-gnu/c++/14/bits/gthr-default.h:713 (streamlogger_utest+0x4aca10) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #26 void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/14/mutex:916 (streamlogger_utest+0x4ea5f1) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #27 std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) /usr/include/c++/14/future:435 (streamlogger_utest+0x4e3f6e) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #28 _M_run /usr/include/c++/14/future:1781 (streamlogger_utest+0x4d85f9) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #29 __invoke_impl<void, void (std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>::*)(), std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>*> /usr/include/c++/14/bits/invoke.h:74 (streamlogger_utest+0x4e2aa5) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #30 __invoke<void (std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>::*)(), std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>*> /usr/include/c++/14/bits/invoke.h:96 (streamlogger_utest+0x4e2760) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #31 _M_invoke<0, 1> /usr/include/c++/14/bits/std_thread.h:301 (streamlogger_utest+0x4e23fc) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #32 operator() /usr/include/c++/14/bits/std_thread.h:308 (streamlogger_utest+0x4e217a) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #33 _M_run /usr/include/c++/14/bits/std_thread.h:253 (streamlogger_utest+0x4e0efa) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #34 <null> <null> (libstdc++.so.6+0xecdb3) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)
6882: 
6882:   Location is heap block of size 40 at 0x720c00000030 allocated by main thread:
6882:     #0 operator new(unsigned long) ../../../../src/libsanitizer/tsan/tsan_new_delete.cpp:64 (libtsan.so.2+0xa6fae) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
6882:     #1 std::__new_allocator<std::_Sp_counted_ptr_inplace<logging::CustomSink, std::allocator<void>, (__gnu_cxx::_Lock_policy)2> >::allocate(unsigned long, void const*) /usr/include/c++/14/bits/new_allocator.h:151 (streamlogger_utest+0x69bad4) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #2 std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<logging::CustomSink, std::allocator<void>, (__gnu_cxx::_Lock_policy)2> > >::allocate(std::allocator<std::_Sp_counted_ptr_inplace<logging::CustomSink, std::allocator<void>, (__gnu_cxx::_Lock_policy)2> >&, unsigned long) /usr/include/c++/14/bits/alloc_traits.h:478 (streamlogger_utest+0x699a35) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #3 std::__allocated_ptr<std::allocator<std::_Sp_counted_ptr_inplace<logging::CustomSink, std::allocator<void>, (__gnu_cxx::_Lock_policy)2> > > std::__allocate_guarded<std::allocator<std::_Sp_counted_ptr_inplace<logging::CustomSink, std::allocator<void>, (__gnu_cxx::_Lock_policy)2> > >(std::allocator<std::_Sp_counted_ptr_inplace<logging::CustomSink, std::allocator<void>, (__gnu_cxx::_Lock_policy)2> >&) /usr/include/c++/14/bits/allocated_ptr.h:98 (streamlogger_utest+0x699a35)
6882:     #4 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<logging::CustomSink, std::allocator<void>>(logging::CustomSink*&, std::_Sp_alloc_shared_tag<std::allocator<void> >) /usr/include/c++/14/bits/shared_ptr_base.h:967 (streamlogger_utest+0x697722) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #5 std::__shared_ptr<logging::CustomSink, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<void>>(std::_Sp_alloc_shared_tag<std::allocator<void> >) /usr/include/c++/14/bits/shared_ptr_base.h:1713 (streamlogger_utest+0x694530) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #6 std::shared_ptr<logging::CustomSink>::shared_ptr<std::allocator<void>>(std::_Sp_alloc_shared_tag<std::allocator<void> >) /usr/include/c++/14/bits/shared_ptr.h:463 (streamlogger_utest+0x691def) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #7 std::shared_ptr<std::enable_if<!std::is_array<logging::CustomSink>::value, logging::CustomSink>::type> std::make_shared<logging::CustomSink>() /usr/include/c++/14/bits/shared_ptr.h:1008 (streamlogger_utest+0x68df21) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #8 logging::start(logging::LoggingConfig const&) /workspaces/Wazuh_5x/wazuh/src/engine/source/base/src/logging.cpp:129 (streamlogger_utest+0x682660) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #9 logging::testInit(logging::Level) /workspaces/Wazuh_5x/wazuh/src/engine/source/base/src/logging.cpp:156 (streamlogger_utest+0x6829fc) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #10 ChannelHandlerTest::SetUp() /workspaces/Wazuh_5x/wazuh/src/engine/source/streamlog/test/src/unit/channel_test.cpp:100 (streamlogger_utest+0x4e6d59) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #11 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /workspaces/Wazuh_5x/wazuh/src/external/googletest/googletest/src/gtest.cc:2607 (streamlogger_utest+0x615d04) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882: 
6882:   Mutex M0 (0x7260000008d8) created at:
6882:     #0 pthread_mutex_lock ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1341 (libtsan.so.2+0x59a13) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
6882:     #1 __gthread_mutex_lock /usr/include/x86_64-linux-gnu/c++/14/bits/gthr-default.h:762 (streamlogger_utest+0x3b37f5) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #2 std::mutex::lock() /usr/include/c++/14/bits/std_mutex.h:113 (streamlogger_utest+0x3bc8a6) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #3 std::lock_guard<std::mutex>::lock_guard(std::mutex&) /usr/include/c++/14/bits/std_mutex.h:250 (streamlogger_utest+0x3cf6ea) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #4 streamlog::ChannelHandler::createWriter() /workspaces/Wazuh_5x/wazuh/src/engine/source/streamlog/src/channel.cpp:681 (streamlogger_utest+0x574370) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #5 operator() /workspaces/Wazuh_5x/wazuh/src/engine/source/streamlog/test/src/unit/channel_test.cpp:957 (streamlogger_utest+0x4bed51) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #6 __invoke_impl<void, ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > /usr/include/c++/14/bits/invoke.h:61 (streamlogger_utest+0x4db71f) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #7 __invoke<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > /usr/include/c++/14/bits/invoke.h:96 (streamlogger_utest+0x4db56e) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #8 _M_invoke<0> /usr/include/c++/14/bits/std_thread.h:301 (streamlogger_utest+0x4db3d6) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #9 operator() /usr/include/c++/14/bits/std_thread.h:308 (streamlogger_utest+0x4db22c) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #10 operator() /usr/include/c++/14/future:1439 (streamlogger_utest+0x4dac27) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #11 __invoke_impl<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>&> /usr/include/c++/14/bits/invoke.h:61 (streamlogger_utest+0x4da5fa) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #12 __invoke_r<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>&> /usr/include/c++/14/bits/invoke.h:114 (streamlogger_utest+0x4d9e49) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #13 _M_invoke /usr/include/c++/14/bits/std_function.h:291 (streamlogger_utest+0x4d95b2) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #14 std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const /usr/include/c++/14/bits/std_function.h:591 (streamlogger_utest+0x4ea6fe) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #15 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) /usr/include/c++/14/future:596 (streamlogger_utest+0x4e40b4) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #16 void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/14/bits/invoke.h:74 (streamlogger_utest+0x4fb9a6) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #17 std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/14/bits/invoke.h:96 (streamlogger_utest+0x4f3a80) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #18 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const /usr/include/c++/14/mutex:909 (streamlogger_utest+0x4ea507) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #19 std::once_flag::_Prepare_execution::_Prepare_execution<std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}>(void (std::__future_base::_State_baseV2::*&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*))::{lambda()#1}::operator()() const /usr/include/c++/14/mutex:845 (streamlogger_utest+0x4f3af6) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #20 std::once_flag::_Prepare_execution::_Prepare_execution<std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}>(void (std::__future_base::_State_baseV2::*&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*))::{lambda()#1}::_FUN() /usr/include/c++/14/mutex:845 (streamlogger_utest+0x4f3b3d) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #21 pthread_once ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1587 (libtsan.so.2+0x57ff5) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
6882:     #22 __gthread_once /usr/include/x86_64-linux-gnu/c++/14/bits/gthr-default.h:713 (streamlogger_utest+0x4aca10) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #23 void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/14/mutex:916 (streamlogger_utest+0x4ea5f1) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #24 std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) /usr/include/c++/14/future:435 (streamlogger_utest+0x4e3f6e) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #25 _M_run /usr/include/c++/14/future:1781 (streamlogger_utest+0x4d85f9) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #26 __invoke_impl<void, void (std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>::*)(), std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>*> /usr/include/c++/14/bits/invoke.h:74 (streamlogger_utest+0x4e2aa5) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #27 __invoke<void (std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>::*)(), std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>*> /usr/include/c++/14/bits/invoke.h:96 (streamlogger_utest+0x4e2760) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #28 _M_invoke<0, 1> /usr/include/c++/14/bits/std_thread.h:301 (streamlogger_utest+0x4e23fc) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #29 operator() /usr/include/c++/14/bits/std_thread.h:308 (streamlogger_utest+0x4e217a) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #30 _M_run /usr/include/c++/14/bits/std_thread.h:253 (streamlogger_utest+0x4e0efa) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #31 <null> <null> (libstdc++.so.6+0xecdb3) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)
6882: 
6882:   Thread T5 (tid=114091, running) created by thread T2 at:
6882:     #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
6882:     #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xeceb0) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)
6882:     #2 streamlog::ChannelHandler::startWorkerThread() /workspaces/Wazuh_5x/wazuh/src/engine/source/streamlog/src/channel.cpp:344 (streamlogger_utest+0x571399) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #3 streamlog::ChannelHandler::createWriter() /workspaces/Wazuh_5x/wazuh/src/engine/source/streamlog/src/channel.cpp:694 (streamlogger_utest+0x5744db) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #4 operator() /workspaces/Wazuh_5x/wazuh/src/engine/source/streamlog/test/src/unit/channel_test.cpp:957 (streamlogger_utest+0x4bed51) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #5 __invoke_impl<void, ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > /usr/include/c++/14/bits/invoke.h:61 (streamlogger_utest+0x4db71f) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #6 __invoke<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > /usr/include/c++/14/bits/invoke.h:96 (streamlogger_utest+0x4db56e) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #7 _M_invoke<0> /usr/include/c++/14/bits/std_thread.h:301 (streamlogger_utest+0x4db3d6) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #8 operator() /usr/include/c++/14/bits/std_thread.h:308 (streamlogger_utest+0x4db22c) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #9 operator() /usr/include/c++/14/future:1439 (streamlogger_utest+0x4dac27) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #10 __invoke_impl<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>&> /usr/include/c++/14/bits/invoke.h:61 (streamlogger_utest+0x4da5fa) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #11 __invoke_r<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>&> /usr/include/c++/14/bits/invoke.h:114 (streamlogger_utest+0x4d9e49) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #12 _M_invoke /usr/include/c++/14/bits/std_function.h:291 (streamlogger_utest+0x4d95b2) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #13 std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const /usr/include/c++/14/bits/std_function.h:591 (streamlogger_utest+0x4ea6fe) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #14 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) /usr/include/c++/14/future:596 (streamlogger_utest+0x4e40b4) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #15 void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/14/bits/invoke.h:74 (streamlogger_utest+0x4fb9a6) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #16 std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/14/bits/invoke.h:96 (streamlogger_utest+0x4f3a80) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #17 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const /usr/include/c++/14/mutex:909 (streamlogger_utest+0x4ea507) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #18 std::once_flag::_Prepare_execution::_Prepare_execution<std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}>(void (std::__future_base::_State_baseV2::*&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*))::{lambda()#1}::operator()() const /usr/include/c++/14/mutex:845 (streamlogger_utest+0x4f3af6) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #19 std::once_flag::_Prepare_execution::_Prepare_execution<std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}>(void (std::__future_base::_State_baseV2::*&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*))::{lambda()#1}::_FUN() /usr/include/c++/14/mutex:845 (streamlogger_utest+0x4f3b3d) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #20 pthread_once ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1587 (libtsan.so.2+0x57ff5) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
6882:     #21 __gthread_once /usr/include/x86_64-linux-gnu/c++/14/bits/gthr-default.h:713 (streamlogger_utest+0x4aca10) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #22 void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/14/mutex:916 (streamlogger_utest+0x4ea5f1) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #23 std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) /usr/include/c++/14/future:435 (streamlogger_utest+0x4e3f6e) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #24 _M_run /usr/include/c++/14/future:1781 (streamlogger_utest+0x4d85f9) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #25 __invoke_impl<void, void (std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>::*)(), std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>*> /usr/include/c++/14/bits/invoke.h:74 (streamlogger_utest+0x4e2aa5) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #26 __invoke<void (std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>::*)(), std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>*> /usr/include/c++/14/bits/invoke.h:96 (streamlogger_utest+0x4e2760) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #27 _M_invoke<0, 1> /usr/include/c++/14/bits/std_thread.h:301 (streamlogger_utest+0x4e23fc) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #28 operator() /usr/include/c++/14/bits/std_thread.h:308 (streamlogger_utest+0x4e217a) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #29 _M_run /usr/include/c++/14/bits/std_thread.h:253 (streamlogger_utest+0x4e0efa) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #30 <null> <null> (libstdc++.so.6+0xecdb3) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)
6882: 
6882:   Thread T4 (tid=114090, running) created by main thread at:
6882:     #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
6882:     #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xeceb0) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)
6882:     #2 _Async_state_impl<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > /usr/include/c++/14/future:1763 (streamlogger_utest+0x4d7d03) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #3 _Construct<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>, ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > /usr/include/c++/14/bits/stl_construct.h:119 (streamlogger_utest+0x4d723e) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #4 construct<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>, ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > /usr/include/c++/14/bits/alloc_traits.h:657 (streamlogger_utest+0x4d57ea) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #5 _Sp_counted_ptr_inplace<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > /usr/include/c++/14/bits/shared_ptr_base.h:607 (streamlogger_utest+0x4d57ea)
6882:     #6 __shared_count<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>, std::allocator<void>, ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > /usr/include/c++/14/bits/shared_ptr_base.h:969 (streamlogger_utest+0x4d4735) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #7 __shared_ptr<std::allocator<void>, ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > /usr/include/c++/14/bits/shared_ptr_base.h:1713 (streamlogger_utest+0x4d3cbf) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #8 shared_ptr<std::allocator<void>, ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > /usr/include/c++/14/bits/shared_ptr.h:463 (streamlogger_utest+0x4d34d8) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #9 make_shared<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > >, void>, ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > /usr/include/c++/14/bits/shared_ptr.h:1008 (streamlogger_utest+0x4d2e7c) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #10 async<ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody()::<lambda()> > /usr/include/c++/14/future:1812 (streamlogger_utest+0x4d26d5) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #11 ChannelHandlerTest_ConcurrentMultipleChannels_Test::TestBody() /workspaces/Wazuh_5x/wazuh/src/engine/source/streamlog/test/src/unit/channel_test.cpp:954 (streamlogger_utest+0x4bf124) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882:     #12 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /workspaces/Wazuh_5x/wazuh/src/external/googletest/googletest/src/gtest.cc:2607 (streamlogger_utest+0x615d04) (BuildId: 93aaf8854262f4c48e8821021932957ab2503b7a)
6882: 
6882: SUMMARY: ThreadSanitizer: data race /workspaces/Wazuh_5x/wazuh/src/engine/source/base/src/logging.cpp:37 in logging::CustomSink::log(spdlog::details::log_msg const&)
6882: ==================

List of UnitTest Failed 

The following tests FAILED:
        6494 - KVDB_Manager_Unit.ParallelColdRace_EventualConvergence (Failed)
        6859 - ChannelHandlerTest.BasicCreationAndDestruction (Failed)
        6866 - ChannelHandlerTest.WriterLifecycle (Failed)
        6867 - ChannelHandlerTest.BasicMessageWriting (Failed)
        6869 - ChannelHandlerTest.TimeBasedRotation (Failed)
        6872 - ChannelHandlerTest.WorkerThreadLifecycle (Failed)
        6874 - ChannelHandlerTest.PlaceholderReplacement (Failed)
        6875 - ChannelHandlerTest.CounterPlaceholderWithSizeRotation (Failed)
        6876 - ChannelHandlerTest.WriterNonCopyable (Failed)
        6877 - ChannelHandlerTest.BufferOverflowBehavior (Failed)
        6878 - ChannelHandlerTest.DirectoryCreation (Failed)
        6880 - ChannelHandlerTest.RegexReplacementEdgeCases (Failed)
        6882 - ChannelHandlerTest.ConcurrentMultipleChannels (Failed)
        6884 - ChannelHandlerTest.MaxSizeBoundaryConditions (Failed)
        6885 - ChannelHandlerTest.PatternValidationEdgeCases (Failed)
        6886 - ChannelHandlerTest.ThreadInterruptionAndCleanup (Failed)
        6887 - ChannelHandlerTest.MemoryPressureTest (Failed)
        6888 - ChannelHandlerTest.ExistingFileResumption (Failed)
        6889 - ChannelHandlerTest.ExistingFileWithRotation (Failed)
        6891 - ChannelHandlerTest.EmptyExistingFileResumption (Failed)
        6892 - ChannelHandlerTest.ConcurrentWritesToExistingFile (Failed)
        6893 - ChannelHandlerTest.LargeExistingFileResumption (Failed)
        6894 - ChannelHandlerTest.FileAppendPositioning (Failed)
        6895 - ChannelHandlerTest.LongChannelName (Failed)
        6898 - ChannelHandlerTest.CompressionWithMockScheduler (Failed)
        6901 - ChannelHandlerTest.CompressionWithDifferentLevels (Failed)
        6902 - ChannelHandlerTest.CompressionWithTimeRotation (Failed)
        6906 - ChannelHandlerTest.StorePersistenceSaveDuringRotation (Failed)
        6908 - ChannelHandlerTest.StorePersistenceSaveError (Failed)
        6921 - Patterns/PatternTest.DifferentPatterns/"logs/${YYYY}/${MM}/${name}-${DD}.json" (Failed)
        6926 - BufferSizes/BufferSizeTest.DifferentBufferSizes/1048576 (Failed)
        7065 - WIndexerConnectorTest.ConcurrentIndexingAndShutdown (Failed)
Errors while running CTest

Race Condition

  1. CustomSink::log() - Concurrent writes at 0x720c0000004c
  2. CustomSink::flush() - Concurrent writes at same address
  3. spdlog::pattern_formatter - Triggered by unsafe CustomSink usage

Root cause: CustomSink lacks synchronization despite being called from multiple threads.
Solution: Made CustomSink thread-safe by inheriting from spdlog::sinks::base_sink<std::mutex> and implementing synchronized methods:

  • Protected sink_it_() for logging
  • Protected flush_() for flushing
  • Mutex automatically managed by base class

Proposed Changes

Issue 1:

  1. Fix lock type: Change cleanup() to use unique_lock for exclusive write access
  2. Add empty validation: Check m_routerWorkers.empty() before accessing front()
  3. Atomic shutdown flag: Add m_isShutdown for thread-safe early exit
  4. Proper lock scoping: Move TOCTOU-vulnerable checks inside lock scope

Affected Methods

  • Orchestrator::getEntries() - line 449
  • Orchestrator::getEntry() - line 408
  • Orchestrator::dumpRouters() - line 120
  • Orchestrator::cleanup() - line 343

Issue 2:

Files Modified:

  • src/engine/source/router/src/orchestrator.cpp
  • src/engine/source/router/include/router/orchestrator.hpp

Key Changes:

  1. Created internal dump methods (no lock acquisition):

    • dumpTestersInternal()
    • dumpRoutersInternal()
    • dumpEpsInternal()

  2. Modified public dump methods to acquire lock and delegate to internal variants

  3. Updated all call sites to use *Internal() when lock already held:

    • postEntry()→ dumpRoutersInternal()
    • deleteEntry() → dumpRoutersInternal()
    • changeEntryPriority() → dumpRoutersInternal()
    • postTestEntry() → dumpTestersInternal()
    • deleteTestEntry() → dumpTestersInternal()
      -renameTestEntry() → dumpTestersInternal()
    • stop() → dumpTestersInternal()
    • loadEpsCounter() → dumpEpsInternal() (constructor context)

Issue 3:

Files Modified:

  • src/engine/source/base/src/logging.cpp
  • src/engine/source/base/include/base/logging.hpp

Key Changes:

  1. Changed CustomSink to inherit from spdlog::sinks::base_sink<std::mutex>
  2. Renamed and protected virtual methods:
    • log() → sink_it_() (protected, synchronized by base class)
    • flush() → flush_() (protected, synchronized by base class)
  3. Removed manual synchronization (handled by base class mutex)
  4. Ensured thread-safety for all concurrent logging operations

⚠️ TSAN note (KVDB / OpenSSL “ColdRace” warning)

While running the KVDB TSAN tests, we still see a ThreadSanitizer data-race warning reported inside OpenSSL (ossl_ht_get / ossl_ht_insert) during the first concurrent SHA256 usage.

KVDB Test Failed 

==================
WARNING: ThreadSanitizer: data race (pid=148453)
  Read of size 8 at 0x721c0001fd68 by thread T9:
    #0 memcmp ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:844 (libtsan.so.2+0x5e1f0) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
    #1 memcmp ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:840 (libtsan.so.2+0x5e1f0)
    #2 ossl_ht_get <null> (kvdbstore_ctest+0x74df8a) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #3 cm::store::dataType::KVDB::updateHash() <null> (kvdbstore_ctest+0x3b2154) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #4 cm::store::dataType::KVDB::KVDB(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, json::Json&&, bool) <null> (kvdbstore_ctest+0x3b235f) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #5 cm::store::dataType::KVDB::fromJson(json::Json const&) <null> (kvdbstore_ctest+0x3b2842) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #6 operator() /workspaces/Wazuh_5x/wazuh/src/engine/source/kvdbstore/test/src/component/kvdb_test.cpp:181 (kvdbstore_ctest+0x3aa140) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #7 __invoke_impl<cm::store::dataType::KVDB, KVDB_Component_ConcurrentColdRace_EventualConvergence_Test::TestBody()::<lambda(const std::string&)>&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&> /usr/include/c++/14/bits/invoke.h:61 (kvdbstore_ctest+0x3ae7e3) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #8 __invoke_r<cm::store::dataType::KVDB, KVDB_Component_ConcurrentColdRace_EventualConvergence_Test::TestBody()::<lambda(const std::string&)>&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&> /usr/include/c++/14/bits/invoke.h:116 (kvdbstore_ctest+0x3ae586) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #9 _M_invoke /usr/include/c++/14/bits/std_function.h:291 (kvdbstore_ctest+0x3ae313) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #10 std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x3ff687) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #11 decltype (((forward<std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&>)({parm#1}))((get<0ul>)((forward<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> >)({parm#2})))) testing::internal::ApplyImpl<std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, 0ul>(std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, testing::internal::IndexSequence<0ul>) <null> (kvdbstore_ctest+0x3fd097) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #12 decltype (ApplyImpl((forward<std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&>)({parm#1}), (forward<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> >)({parm#2}), (testing::internal::MakeIndexSequenceImpl<std::tuple_size<std::remove_reference<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> >::type>::value>::type)())) testing::internal::Apply<std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> >(std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&) <null> (kvdbstore_ctest+0x3f6792) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #13 testing::Action<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::Perform(std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>) const <null> (kvdbstore_ctest+0x3ed671) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #14 testing::internal::FunctionMocker<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::PerformDefaultAction(std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x3ed399) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #15 testing::internal::ActionResultHolder<cm::store::dataType::KVDB>* testing::internal::ActionResultHolder<cm::store::dataType::KVDB>::PerformDefaultAction<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>(testing::internal::FunctionMocker<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const*, testing::internal::Function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::ArgumentTuple&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (kvdbstore_ctest+0x3e57b8) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #16 testing::internal::FunctionMocker<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::UntypedPerformDefaultAction(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x3e0910) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #17 testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith(void*) /workspaces/Wazuh_5x/wazuh/src/external/googletest/googlemock/src/gmock-spec-builders.cc:409 (kvdbstore_ctest+0x4070b4) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #18 cm::store::MockICMStoreNSReader::getKVDBByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x3b4ccb) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #19 auto cm::store::ICMStoreNSReader::getResourceByName<cm::store::dataType::KVDB>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x417862) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #20 kvdbstore::KVDBManager::getKVDBHandler(cm::store::ICMStoreNSReader const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /workspaces/Wazuh_5x/wazuh/src/engine/source/kvdbstore/src/kvdbManager.cpp:36 (kvdbstore_ctest+0x416a59) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #21 operator() /workspaces/Wazuh_5x/wazuh/src/engine/source/kvdbstore/test/src/component/kvdb_test.cpp:42 (kvdbstore_ctest+0x3a60da) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #22 __invoke_impl<void, (anonymous namespace)::parallelReadRefs(kvdbstore::KVDBManager&, const cm::store::ICMStoreNSReader&, const std::string&, const std::string&, int)::<lambda()> > /usr/include/c++/14/bits/invoke.h:61 (kvdbstore_ctest+0x3af600) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #23 __invoke<(anonymous namespace)::parallelReadRefs(kvdbstore::KVDBManager&, const cm::store::ICMStoreNSReader&, const std::string&, const std::string&, int)::<lambda()> > /usr/include/c++/14/bits/invoke.h:96 (kvdbstore_ctest+0x3af577) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #24 _M_invoke<0> /usr/include/c++/14/bits/std_thread.h:301 (kvdbstore_ctest+0x3af4d8) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #25 operator() /usr/include/c++/14/bits/std_thread.h:308 (kvdbstore_ctest+0x3af47e) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #26 _M_run /usr/include/c++/14/bits/std_thread.h:253 (kvdbstore_ctest+0x3af434) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #27 <null> <null> (libstdc++.so.6+0xecdb3) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)

  Previous write of size 8 at 0x721c0001fd68 by thread T16 (mutexes: write M0):
    #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc:115 (libtsan.so.2+0x8bd30) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
    #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc:107 (libtsan.so.2+0x8bd30)
    #2 ossl_ht_insert <null> (kvdbstore_ctest+0x74ddd6) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #3 cm::store::dataType::KVDB::updateHash() <null> (kvdbstore_ctest+0x3b2154) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #4 cm::store::dataType::KVDB::KVDB(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, json::Json&&, bool) <null> (kvdbstore_ctest+0x3b235f) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #5 cm::store::dataType::KVDB::fromJson(json::Json const&) <null> (kvdbstore_ctest+0x3b2842) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #6 operator() /workspaces/Wazuh_5x/wazuh/src/engine/source/kvdbstore/test/src/component/kvdb_test.cpp:181 (kvdbstore_ctest+0x3aa140) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #7 __invoke_impl<cm::store::dataType::KVDB, KVDB_Component_ConcurrentColdRace_EventualConvergence_Test::TestBody()::<lambda(const std::string&)>&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&> /usr/include/c++/14/bits/invoke.h:61 (kvdbstore_ctest+0x3ae7e3) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #8 __invoke_r<cm::store::dataType::KVDB, KVDB_Component_ConcurrentColdRace_EventualConvergence_Test::TestBody()::<lambda(const std::string&)>&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&> /usr/include/c++/14/bits/invoke.h:116 (kvdbstore_ctest+0x3ae586) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #9 _M_invoke /usr/include/c++/14/bits/std_function.h:291 (kvdbstore_ctest+0x3ae313) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #10 std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x3ff687) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #11 decltype (((forward<std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&>)({parm#1}))((get<0ul>)((forward<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> >)({parm#2})))) testing::internal::ApplyImpl<std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, 0ul>(std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, testing::internal::IndexSequence<0ul>) <null> (kvdbstore_ctest+0x3fd097) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #12 decltype (ApplyImpl((forward<std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&>)({parm#1}), (forward<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> >)({parm#2}), (testing::internal::MakeIndexSequenceImpl<std::tuple_size<std::remove_reference<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> >::type>::value>::type)())) testing::internal::Apply<std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> >(std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&) <null> (kvdbstore_ctest+0x3f6792) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #13 testing::Action<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::Perform(std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>) const <null> (kvdbstore_ctest+0x3ed671) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #14 testing::internal::FunctionMocker<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::PerformDefaultAction(std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x3ed399) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #15 testing::internal::ActionResultHolder<cm::store::dataType::KVDB>* testing::internal::ActionResultHolder<cm::store::dataType::KVDB>::PerformDefaultAction<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>(testing::internal::FunctionMocker<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const*, testing::internal::Function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::ArgumentTuple&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (kvdbstore_ctest+0x3e57b8) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #16 testing::internal::FunctionMocker<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::UntypedPerformDefaultAction(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x3e0910) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #17 testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith(void*) /workspaces/Wazuh_5x/wazuh/src/external/googletest/googlemock/src/gmock-spec-builders.cc:409 (kvdbstore_ctest+0x4070b4) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #18 cm::store::MockICMStoreNSReader::getKVDBByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x3b4ccb) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #19 auto cm::store::ICMStoreNSReader::getResourceByName<cm::store::dataType::KVDB>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x417862) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #20 kvdbstore::KVDBManager::getKVDBHandler(cm::store::ICMStoreNSReader const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /workspaces/Wazuh_5x/wazuh/src/engine/source/kvdbstore/src/kvdbManager.cpp:36 (kvdbstore_ctest+0x416a59) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #21 operator() /workspaces/Wazuh_5x/wazuh/src/engine/source/kvdbstore/test/src/component/kvdb_test.cpp:42 (kvdbstore_ctest+0x3a60da) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #22 __invoke_impl<void, (anonymous namespace)::parallelReadRefs(kvdbstore::KVDBManager&, const cm::store::ICMStoreNSReader&, const std::string&, const std::string&, int)::<lambda()> > /usr/include/c++/14/bits/invoke.h:61 (kvdbstore_ctest+0x3af600) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #23 __invoke<(anonymous namespace)::parallelReadRefs(kvdbstore::KVDBManager&, const cm::store::ICMStoreNSReader&, const std::string&, const std::string&, int)::<lambda()> > /usr/include/c++/14/bits/invoke.h:96 (kvdbstore_ctest+0x3af577) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #24 _M_invoke<0> /usr/include/c++/14/bits/std_thread.h:301 (kvdbstore_ctest+0x3af4d8) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #25 operator() /usr/include/c++/14/bits/std_thread.h:308 (kvdbstore_ctest+0x3af47e) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #26 _M_run /usr/include/c++/14/bits/std_thread.h:253 (kvdbstore_ctest+0x3af434) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #27 <null> <null> (libstdc++.so.6+0xecdb3) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)

  Location is heap block of size 104 at 0x721c0001fd40 allocated by thread T16:
    #0 malloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:665 (libtsan.so.2+0x54b3f) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
    #1 CRYPTO_malloc <null> (kvdbstore_ctest+0x5db990) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #2 cm::store::dataType::KVDB::updateHash() <null> (kvdbstore_ctest+0x3b2154) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #3 cm::store::dataType::KVDB::KVDB(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, json::Json&&, bool) <null> (kvdbstore_ctest+0x3b235f) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #4 cm::store::dataType::KVDB::fromJson(json::Json const&) <null> (kvdbstore_ctest+0x3b2842) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #5 operator() /workspaces/Wazuh_5x/wazuh/src/engine/source/kvdbstore/test/src/component/kvdb_test.cpp:181 (kvdbstore_ctest+0x3aa140) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #6 __invoke_impl<cm::store::dataType::KVDB, KVDB_Component_ConcurrentColdRace_EventualConvergence_Test::TestBody()::<lambda(const std::string&)>&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&> /usr/include/c++/14/bits/invoke.h:61 (kvdbstore_ctest+0x3ae7e3) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #7 __invoke_r<cm::store::dataType::KVDB, KVDB_Component_ConcurrentColdRace_EventualConvergence_Test::TestBody()::<lambda(const std::string&)>&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&> /usr/include/c++/14/bits/invoke.h:116 (kvdbstore_ctest+0x3ae586) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #8 _M_invoke /usr/include/c++/14/bits/std_function.h:291 (kvdbstore_ctest+0x3ae313) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #9 std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x3ff687) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #10 decltype (((forward<std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&>)({parm#1}))((get<0ul>)((forward<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> >)({parm#2})))) testing::internal::ApplyImpl<std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, 0ul>(std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, testing::internal::IndexSequence<0ul>) <null> (kvdbstore_ctest+0x3fd097) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #11 decltype (ApplyImpl((forward<std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&>)({parm#1}), (forward<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> >)({parm#2}), (testing::internal::MakeIndexSequenceImpl<std::tuple_size<std::remove_reference<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> >::type>::value>::type)())) testing::internal::Apply<std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> >(std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&) <null> (kvdbstore_ctest+0x3f6792) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #12 testing::Action<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::Perform(std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>) const <null> (kvdbstore_ctest+0x3ed671) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #13 testing::internal::FunctionMocker<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::PerformDefaultAction(std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x3ed399) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #14 testing::internal::ActionResultHolder<cm::store::dataType::KVDB>* testing::internal::ActionResultHolder<cm::store::dataType::KVDB>::PerformDefaultAction<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>(testing::internal::FunctionMocker<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const*, testing::internal::Function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::ArgumentTuple&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (kvdbstore_ctest+0x3e57b8) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #15 testing::internal::FunctionMocker<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::UntypedPerformDefaultAction(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x3e0910) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #16 testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith(void*) /workspaces/Wazuh_5x/wazuh/src/external/googletest/googlemock/src/gmock-spec-builders.cc:409 (kvdbstore_ctest+0x4070b4) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #17 cm::store::MockICMStoreNSReader::getKVDBByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x3b4ccb) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #18 auto cm::store::ICMStoreNSReader::getResourceByName<cm::store::dataType::KVDB>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x417862) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #19 kvdbstore::KVDBManager::getKVDBHandler(cm::store::ICMStoreNSReader const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /workspaces/Wazuh_5x/wazuh/src/engine/source/kvdbstore/src/kvdbManager.cpp:36 (kvdbstore_ctest+0x416a59) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #20 operator() /workspaces/Wazuh_5x/wazuh/src/engine/source/kvdbstore/test/src/component/kvdb_test.cpp:42 (kvdbstore_ctest+0x3a60da) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #21 __invoke_impl<void, (anonymous namespace)::parallelReadRefs(kvdbstore::KVDBManager&, const cm::store::ICMStoreNSReader&, const std::string&, const std::string&, int)::<lambda()> > /usr/include/c++/14/bits/invoke.h:61 (kvdbstore_ctest+0x3af600) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #22 __invoke<(anonymous namespace)::parallelReadRefs(kvdbstore::KVDBManager&, const cm::store::ICMStoreNSReader&, const std::string&, const std::string&, int)::<lambda()> > /usr/include/c++/14/bits/invoke.h:96 (kvdbstore_ctest+0x3af577) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #23 _M_invoke<0> /usr/include/c++/14/bits/std_thread.h:301 (kvdbstore_ctest+0x3af4d8) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #24 operator() /usr/include/c++/14/bits/std_thread.h:308 (kvdbstore_ctest+0x3af47e) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #25 _M_run /usr/include/c++/14/bits/std_thread.h:253 (kvdbstore_ctest+0x3af434) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #26 <null> <null> (libstdc++.so.6+0xecdb3) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)

  Mutex M0 (0x72100000e500) created at:
    #0 pthread_rwlock_init ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1458 (libtsan.so.2+0x5633d) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
    #1 CRYPTO_THREAD_lock_new <null> (kvdbstore_ctest+0x5e63f9) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #2 CRYPTO_THREAD_run_once <null> (kvdbstore_ctest+0x5e64c8) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #3 CRYPTO_THREAD_run_once <null> (kvdbstore_ctest+0x5e64c8) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #4 cm::store::dataType::KVDB::updateHash() <null> (kvdbstore_ctest+0x3b2154) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #5 cm::store::dataType::KVDB::KVDB(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, json::Json&&, bool) <null> (kvdbstore_ctest+0x3b235f) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #6 cm::store::dataType::KVDB::fromJson(json::Json const&) <null> (kvdbstore_ctest+0x3b2842) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #7 operator() /workspaces/Wazuh_5x/wazuh/src/engine/source/kvdbstore/test/src/component/kvdb_test.cpp:181 (kvdbstore_ctest+0x3aa140) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #8 __invoke_impl<cm::store::dataType::KVDB, KVDB_Component_ConcurrentColdRace_EventualConvergence_Test::TestBody()::<lambda(const std::string&)>&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&> /usr/include/c++/14/bits/invoke.h:61 (kvdbstore_ctest+0x3ae7e3) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #9 __invoke_r<cm::store::dataType::KVDB, KVDB_Component_ConcurrentColdRace_EventualConvergence_Test::TestBody()::<lambda(const std::string&)>&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&> /usr/include/c++/14/bits/invoke.h:116 (kvdbstore_ctest+0x3ae586) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #10 _M_invoke /usr/include/c++/14/bits/std_function.h:291 (kvdbstore_ctest+0x3ae313) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #11 std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x3ff687) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #12 decltype (((forward<std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&>)({parm#1}))((get<0ul>)((forward<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> >)({parm#2})))) testing::internal::ApplyImpl<std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, 0ul>(std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, testing::internal::IndexSequence<0ul>) <null> (kvdbstore_ctest+0x3fd097) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #13 decltype (ApplyImpl((forward<std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&>)({parm#1}), (forward<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> >)({parm#2}), (testing::internal::MakeIndexSequenceImpl<std::tuple_size<std::remove_reference<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> >::type>::value>::type)())) testing::internal::Apply<std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> >(std::function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&) <null> (kvdbstore_ctest+0x3f6792) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #14 testing::Action<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::Perform(std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>) const <null> (kvdbstore_ctest+0x3ed671) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #15 testing::internal::FunctionMocker<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::PerformDefaultAction(std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x3ed399) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #16 testing::internal::ActionResultHolder<cm::store::dataType::KVDB>* testing::internal::ActionResultHolder<cm::store::dataType::KVDB>::PerformDefaultAction<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>(testing::internal::FunctionMocker<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const*, testing::internal::Function<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::ArgumentTuple&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (kvdbstore_ctest+0x3e57b8) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #17 testing::internal::FunctionMocker<cm::store::dataType::KVDB (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::UntypedPerformDefaultAction(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x3e0910) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #18 testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith(void*) /workspaces/Wazuh_5x/wazuh/src/external/googletest/googlemock/src/gmock-spec-builders.cc:409 (kvdbstore_ctest+0x4070b4) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #19 cm::store::MockICMStoreNSReader::getKVDBByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x3b4ccb) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #20 auto cm::store::ICMStoreNSReader::getResourceByName<cm::store::dataType::KVDB>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const <null> (kvdbstore_ctest+0x417862) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #21 kvdbstore::KVDBManager::getKVDBHandler(cm::store::ICMStoreNSReader const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /workspaces/Wazuh_5x/wazuh/src/engine/source/kvdbstore/src/kvdbManager.cpp:36 (kvdbstore_ctest+0x416a59) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #22 operator() /workspaces/Wazuh_5x/wazuh/src/engine/source/kvdbstore/test/src/component/kvdb_test.cpp:42 (kvdbstore_ctest+0x3a60da) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #23 __invoke_impl<void, (anonymous namespace)::parallelReadRefs(kvdbstore::KVDBManager&, const cm::store::ICMStoreNSReader&, const std::string&, const std::string&, int)::<lambda()> > /usr/include/c++/14/bits/invoke.h:61 (kvdbstore_ctest+0x3af600) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #24 __invoke<(anonymous namespace)::parallelReadRefs(kvdbstore::KVDBManager&, const cm::store::ICMStoreNSReader&, const std::string&, const std::string&, int)::<lambda()> > /usr/include/c++/14/bits/invoke.h:96 (kvdbstore_ctest+0x3af577) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #25 _M_invoke<0> /usr/include/c++/14/bits/std_thread.h:301 (kvdbstore_ctest+0x3af4d8) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #26 operator() /usr/include/c++/14/bits/std_thread.h:308 (kvdbstore_ctest+0x3af47e) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #27 _M_run /usr/include/c++/14/bits/std_thread.h:253 (kvdbstore_ctest+0x3af434) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #28 <null> <null> (libstdc++.so.6+0xecdb3) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)

  Thread T9 (tid=148463, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xeceb0) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)
    #2 construct<std::thread, (anonymous namespace)::parallelReadRefs(kvdbstore::KVDBManager&, const cm::store::ICMStoreNSReader&, const std::string&, const std::string&, int)::<lambda()> > /usr/include/c++/14/bits/new_allocator.h:191 (kvdbstore_ctest+0x3ad8f5) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #3 construct<std::thread, (anonymous namespace)::parallelReadRefs(kvdbstore::KVDBManager&, const cm::store::ICMStoreNSReader&, const std::string&, const std::string&, int)::<lambda()> > /usr/include/c++/14/bits/alloc_traits.h:534 (kvdbstore_ctest+0x3ad8f5)
    #4 emplace_back<(anonymous namespace)::parallelReadRefs(kvdbstore::KVDBManager&, const cm::store::ICMStoreNSReader&, const std::string&, const std::string&, int)::<lambda()> > /usr/include/c++/14/bits/vector.tcc:117 (kvdbstore_ctest+0x3ad8f5)
    #5 parallelReadRefs /workspaces/Wazuh_5x/wazuh/src/engine/source/kvdbstore/test/src/component/kvdb_test.cpp:35 (kvdbstore_ctest+0x3a6543) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #6 KVDB_Component_ConcurrentColdRace_EventualConvergence_Test::TestBody() /workspaces/Wazuh_5x/wazuh/src/engine/source/kvdbstore/test/src/component/kvdb_test.cpp:186 (kvdbstore_ctest+0x3aa4bf) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #7 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /workspaces/Wazuh_5x/wazuh/src/external/googletest/googletest/src/gtest.cc:2607 (kvdbstore_ctest+0x461202) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)

  Thread T16 (tid=148470, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xeceb0) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)
    #2 construct<std::thread, (anonymous namespace)::parallelReadRefs(kvdbstore::KVDBManager&, const cm::store::ICMStoreNSReader&, const std::string&, const std::string&, int)::<lambda()> > /usr/include/c++/14/bits/new_allocator.h:191 (kvdbstore_ctest+0x3ad8f5) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #3 construct<std::thread, (anonymous namespace)::parallelReadRefs(kvdbstore::KVDBManager&, const cm::store::ICMStoreNSReader&, const std::string&, const std::string&, int)::<lambda()> > /usr/include/c++/14/bits/alloc_traits.h:534 (kvdbstore_ctest+0x3ad8f5)
    #4 emplace_back<(anonymous namespace)::parallelReadRefs(kvdbstore::KVDBManager&, const cm::store::ICMStoreNSReader&, const std::string&, const std::string&, int)::<lambda()> > /usr/include/c++/14/bits/vector.tcc:117 (kvdbstore_ctest+0x3ad8f5)
    #5 parallelReadRefs /workspaces/Wazuh_5x/wazuh/src/engine/source/kvdbstore/test/src/component/kvdb_test.cpp:35 (kvdbstore_ctest+0x3a6543) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #6 KVDB_Component_ConcurrentColdRace_EventualConvergence_Test::TestBody() /workspaces/Wazuh_5x/wazuh/src/engine/source/kvdbstore/test/src/component/kvdb_test.cpp:186 (kvdbstore_ctest+0x3aa4bf) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)
    #7 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /workspaces/Wazuh_5x/wazuh/src/external/googletest/googletest/src/gtest.cc:2607 (kvdbstore_ctest+0x461202) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789)

SUMMARY: ThreadSanitizer: data race (/workspaces/Wazuh_5x/wazuh/src/build/engine/source/kvdbstore/kvdbstore_ctest+0x74df8a) (BuildId: 83c0fcbe67cdba68ee956f6746ecdc2910fef789) in ossl_ht_get
==================


99% tests passed, 2 tests failed out of 158

Total Test time (real) =  35.60 sec

The following tests FAILED:
	6494 - KVDB_Manager_Unit.ParallelColdRace_EventualConvergence (Failed)
	6501 - KVDB_Component.ConcurrentColdRace_EventualConvergence (Failed)
Errors while running CTest

This is triggered by KVDB::updateHash() → base::utils::hash::sha256() (each call creates its own EVP_MD_CTX, so we’re not sharing hash contexts or KVDB state across threads).

Based on the investigation, this does not look like a KVDB bug (no shared Wazuh memory involved). It matches an OpenSSL 3.x lazy-init / cold-start behavior where internal registries/caches are initialized when multiple threads call SHA256 at the same time

Options to handle it:

  1. Accept & document it as a benign/upstream TSAN warning (OpenSSL internal init).
  2. Suppress it in TSAN (CI/workflow) targeting only these symbols (ossl_ht_get / ossl_ht_insert), so we don’t hide unrelated races.
  3. Warm-up OpenSSL before the parallel section (e.g., a single early SHA256 call during init / test setup) to avoid concurrent first-time initialization.

No KVDB functional failures were observed; this is a TSAN signal coming from OpenSSL internals during cold-start concurrency.

UPDATE: We first tried to avoid the OpenSSL “cold-start” TSAN race by serializing the KVDB TSAN tests with gtest_discover_tests(... PROPERTIES RESOURCE_LOCK ...) (same idea as wIndexerConnector_utest), but it didn’t help—the OpenSSL warning still showed up.

Given that, we went with the warm-up approach instead: we added a one-time SHA256 call during the test suite setup to force OpenSSL initialization before the parallel section.

To apply this with minimal code changes, we updated these test fixtures:

  • KVDB_Component (in kvdb_test.cpp)
  • KVDB_Manager_Unit (in kvdb_manager_test.cpp)

Both now include a static void SetUpTestSuite() that does the warm-up SHA256 call, and the affected tests were switched from TEST(...) to TEST_F(...) so they run under the fixture setup.

Results and Evidence

Ran TSAN build multiple times - no crashes observed after fix. Previously reproduced consistently within 2-3 runs.

@jam300 jam300 linked an issue Dec 19, 2025 that may be closed by this pull request
Spike: Analyze TSAN-report of wazuh-engine in test mode #32387
Closed
@jam300 jam300 self-assigned this Dec 19, 2025
@jam300 jam300 marked this pull request as draft December 19, 2025 06:08
@jam300 jam300 marked this pull request as ready for review December 19, 2025 06:08
matigarciadev
matigarciadev reviewed Dec 22, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@matigarciadev matigarciadev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! But I'd like us to analyze this:

Compiled with TSAN + Debug and ran the unit tests from:
/workspaces/wazuh-5.x/wazuh/src

Commands used:

  • Parallel run:
    ctest --test-dir build --output-on-failure -j "$(nproc)"
  • Single-worker run:
    ctest --test-dir build --output-on-failure -j 1

In both cases we saw failures in a very similar set of tests.

Failed tests (from the TSAN build):

  • 6859 - ChannelHandlerTest.BasicCreationAndDestruction
  • 6866 - ChannelHandlerTest.WriterLifecycle
  • 6867 - ChannelHandlerTest.BasicMessageWriting
  • 6868 - ChannelHandlerTest.SizeBasedRotation
  • 6869 - ChannelHandlerTest.TimeBasedRotation
  • 6870 - ChannelHandlerTest.HardLinkCreation
  • 6871 - ChannelHandlerTest.ConcurrentWriters
  • 6872 - ChannelHandlerTest.WorkerThreadLifecycle
  • 6874 - ChannelHandlerTest.PlaceholderReplacement
  • 6875 - ChannelHandlerTest.CounterPlaceholderWithSizeRotation
  • 6876 - ChannelHandlerTest.WriterNonCopyable
  • 6877 - ChannelHandlerTest.BufferOverflowBehavior
  • 6878 - ChannelHandlerTest.DirectoryCreation
  • 6880 - ChannelHandlerTest.RegexReplacementEdgeCases
  • 6881 - ChannelHandlerTest.ExtremelyLongMessages
  • 6882 - ChannelHandlerTest.ConcurrentMultipleChannels
  • 6884 - ChannelHandlerTest.MaxSizeBoundaryConditions
  • 6885 - ChannelHandlerTest.PatternValidationEdgeCases
  • 6886 - ChannelHandlerTest.ThreadInterruptionAndCleanup
  • 6887 - ChannelHandlerTest.MemoryPressureTest
  • 6888 - ChannelHandlerTest.ExistingFileResumption
  • 6889 - ChannelHandlerTest.ExistingFileWithRotation
  • 6890 - ChannelHandlerTest.FileSizeTrackingAccuracy
  • 6891 - ChannelHandlerTest.EmptyExistingFileResumption
  • 6892 - ChannelHandlerTest.ConcurrentWritesToExistingFile
  • 6893 - ChannelHandlerTest.LargeExistingFileResumption
  • 6894 - ChannelHandlerTest.FileAppendPositioning
  • 6895 - ChannelHandlerTest.LongChannelName
  • 6898 - ChannelHandlerTest.CompressionWithMockScheduler
  • 6899 - ChannelHandlerTest.CompressionWithoutScheduler
  • 6901 - ChannelHandlerTest.CompressionWithDifferentLevels
  • 6902 - ChannelHandlerTest.CompressionWithTimeRotation
  • 6906 - ChannelHandlerTest.StorePersistenceSaveDuringRotation
  • 6908 - ChannelHandlerTest.StorePersistenceSaveError
  • 6920 - Patterns/PatternTest.DifferentPatterns/"${YYYY}-${MM}-${DD}-${name}.log"
  • 6921 - Patterns/PatternTest.DifferentPatterns/"logs/${YYYY}/${MM}/${name}-${DD}.json"
  • 6923 - BufferSizes/BufferSizeTest.DifferentBufferSizes/1
  • 6924 - BufferSizes/BufferSizeTest.DifferentBufferSizes/10
  • 6925 - BufferSizes/BufferSizeTest.DifferentBufferSizes/1024
  • 7065 - WIndexerConnectorTest.ConcurrentIndexingAndShutdown

However, when running the test binaries directly (e.g. ./streamlogger_utest) the suites completed successfully and we couldn’t reproduce the failures that show up under ctest.

After that, I cleaned the build and recompiled without TSAN. With the non-TSAN build I couldn’t reproduce any of these errors, regardless of running via ctest with -j "$(nproc)" or -j 1.

What i think is happening: when tests are executed via ctest, the execution model/working directory can differ from running the binary directly (often ctest ends up invoking the same gtest executable multiple times with --gtest_filter=..., i.e. one process per test case). Combined with TSAN’s overhead/timing changes, this can surface races or shared-resource assumptions (filesystem paths, temp dirs, global state) that don’t show up when the full suite runs in a single process.

@jam300 jam300 force-pushed the bug/32387-Analyze_TSAN branch 2 times, most recently from e54348b to 3f9a7c9 Compare December 24, 2025 00:05
@jam300 jam300 requested a review from matigarciadev December 24, 2025 14:33
@jam300 jam300 force-pushed the bug/32387-Analyze_TSAN branch from 3f9a7c9 to 4dccd14 Compare December 24, 2025 14:37
@matigarciadev
Copy link
Copy Markdown
Member

matigarciadev commented Dec 26, 2025

I ran the tests again with the changes we implemented and observed the following:

The following tests FAILED:
        6501 - KVDB_Manager_Unit.ParallelColdRace_EventualConvergence (Failed)
        6508 - KVDB_Component.ConcurrentColdRace_EventualConvergence (Failed)

LucioDonda
LucioDonda requested changes Dec 26, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@LucioDonda LucioDonda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Built and Install
General settings:
    TARGET:             server
    V:                  
    DEBUG:              
    INSTALLDIR:         /var/ossec
    DATABASE:           
    ONEWAY:             no
    CLEANFULL:          no
    RESOURCES_URL:      https://packages.wazuh.com/deps/99-29585
    EXTERNAL_SRC_ONLY:  
    HTTP_REQUEST_BRANCH:cd50797cfe03c27f3759bdc243fecca6f7535d35
User settings:
    WAZUH_GROUP:        wazuh
    WAZUH_USER:         wazuh
USE settings:
    USE_INOTIFY:        no
    USE_BIG_ENDIAN:     no
    USE_SELINUX:        no
    USE_AUDIT:          yes
    DISABLE_SYSC:       no
    IMAGE_TRUST_CHECKS: 1
    CA_NAME:            DigiCert Assured ID Root CA
Mysql settings:
    includes:           
    libs:               
Pgsql settings:
    includes:           
    libs:               
Defines:
    -DOSSECHIDS -DUSER="wazuh" -DGROUPGLOBAL="wazuh" -DLinux -DINOTIFY_ENABLED -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DIMAGE_TRUST_CHECKS=1 -DCA_NAME='DigiCert Assured ID Root CA' -DENABLE_SYSC -DENABLE_AUDIT
Compiler:
    CFLAGS            -pthread -Iexternal/pacman/lib/libalpm/ -Iexternal/libarchive/libarchive -Wl,--start-group -Iexternal/audit-userspace/lib -g -DNDEBUG -O2 -DOSSECHIDS -DUSER="wazuh" -DGROUPGLOBAL="wazuh" -DLinux -DINOTIFY_ENABLED -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DIMAGE_TRUST_CHECKS=1 -DCA_NAME='DigiCert Assured ID Root CA' -DENABLE_SYSC -DENABLE_AUDIT -pipe -Wall -Wextra -std=gnu99 -I./ -I./headers/ -Iexternal/openssl/include -Iexternal/cJSON/ -Iexternal/libyaml/include -Iexternal/curl/include -Iexternal/msgpack/include -Iexternal/bzip2/ -Ishared_modules/common -Ishared_modules/dbsync/include -Ishared_modules/sync_protocol/include -Iwazuh_modules/syscollector/include -Iwazuh_modules/sca/include -Iwazuh_modules/agent_info/include -Idata_provider/include -Iexternal/libpcre2/include -Iexternal/rpm//builddir/output/include -Isyscheckd/include -Ishared_modules/router/include -Ishared_modules/content_manager/include -Ishared_modules/file_helper/file_io/include -Ishared_modules/file_helper/filesystem/include -Iwazuh_modules/vulnerability_scanner/include -I./shared_modules/ 
    LDFLAGS           '-Wl,-rpath,/../lib' -pthread -lrt -ldl -O2 -Lshared_modules/dbsync/build/lib -Lshared_modules/sync_protocol/build/lib -Lshared_modules/file_helper/build/lib  -Lwazuh_modules/syscollector/build/lib -Lwazuh_modules/sca/build/lib -Lwazuh_modules/agent_info/build/lib -Ldata_provider/build/lib -Lsyscheckd/build/lib
    LIBS              -lrt -ldl -lm 
    CC                gcc
    MAKE              make
make[1]: Leaving directory '/workspaces/wazuh-5.x-v1/wazuh/src'

Done building server

Wait for success...
success
Makefile:927: warning: overriding recipe for target 'external/simdjson/build/libsimdjson.a'
Makefile:898: warning: ignoring old recipe for target 'external/simdjson/build/libsimdjson.a'
Makefile:2390: warning: overriding recipe for target 'win32/ui_resource.o'
Makefile:2333: warning: ignoring old recipe for target 'win32/ui_resource.o'
mkdir -p /var/ossec/framework/python
cp external/cpython.tar.gz /var/ossec/framework/python/cpython.tar.gz && tar -xf /var/ossec/framework/python/cpython.tar.gz -C /var/ossec/framework/python && rm -rf /var/ossec/framework/python/cpython.tar.gz
find /var/ossec/framework/python -name "*libpython3.10.so.1.0" -exec ln -f {} /var/ossec/lib/libpython3.10.so.1.0 \;
cd ../framework && /var/ossec/framework/python/bin/python3 -m pip install . --use-pep517 --prefix=/var/ossec/framework/python && rm -rf build/
Processing /workspaces/wazuh-5.x-v1/wazuh/framework
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: wazuh
  Building wheel for wazuh (pyproject.toml) ... done
  Created wheel for wazuh: filename=wazuh-5.0.0-py3-none-any.whl size=263526 sha256=2fb35839b8d88fd84f9373d3bf820da06c9b9bb5992385188ac7728ebd550099
  Stored in directory: /tmp/pip-ephem-wheel-cache-x1115zu2/wheels/4d/02/17/3d166d40b1fe4e9716dffb3ec7f332f2cfdbea9c6e1af216c7
Successfully built wazuh
Installing collected packages: wazuh
Successfully installed wazuh-5.0.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.3.2 -> 25.3
[notice] To update, run: /var/ossec/framework/python/bin/python3 -m pip install --upgrade pip
chown -R root:wazuh /var/ossec/framework/python
chmod -R o=- /var/ossec/framework/python
cd ../api && /var/ossec/framework/python/bin/python3 -m pip install . --use-pep517 --prefix=/var/ossec/framework/python && rm -rf build/
Processing /workspaces/wazuh-5.x-v1/wazuh/api
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: api
  Building wheel for api (pyproject.toml) ... done
  Created wheel for api: filename=api-5.0.0-py3-none-any.whl size=108629 sha256=6b8dc98775c87737dcba003356baf8931a90f207e03b105d916e0569d4892e85
  Stored in directory: /tmp/pip-ephem-wheel-cache-riitv6zr/wheels/bd/95/a9/30937339a19727bbd24a28690efa0ebac097f7d0a4f02c4ceb
Successfully built api
Installing collected packages: api
Successfully installed api-5.0.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.3.2 -> 25.3
[notice] To update, run: /var/ossec/framework/python/bin/python3 -m pip install --upgrade pip
Generating schema files...
Loading resources...
Loading WCS files from directory external/wcs-flat-files/...
Found 6 YAML files to merge: ['applications-ecs-flat.yml', 'system-activity-ecs-flat.yml', 'security-ecs-flat.yml', 'access-management-ecs-flat.yml', 'other-ecs-flat.yml', 'network-activity-ecs-flat.yml']
Loading access-management-ecs-flat.yml...
Loading applications-ecs-flat.yml...
Loading network-activity-ecs-flat.yml...
Loading other-ecs-flat.yml...
Loading security-ecs-flat.yml...
Loading system-activity-ecs-flat.yml...
Successfully merged 6 files into temporary file: /tmp/merged_wcs_9v11e0hc.yml
Loading schema template...
Loading logpar overrides template...
Building field tree from WCS definition...
Success.
Generating engine schema...
Generating fields schema properties...
Success.
Generating clean properties mapping...
Success.
Generating logpar configuration...
Success.
Cleaned up temporary file: /tmp/merged_wcs_9v11e0hc.yml
Saving files to "engine/ruleset/schemas/"...
Success.
Schema files generated successfully.
Copying store files...
Engine store installed successfully.
Engine output configuration files installed successfully.
Removing old SCA policies...
Installing SCA policies...
Installing additional SCA policies...
Generating self-signed certificate for wazuh-authd...


 - System is Debian (Ubuntu or derivative).
 - Init script modified to start Wazuh during boot.
Starting Wazuh...
server
2025/12/26 11:54:47 wazuh-modulesd: ERROR: File '/var/ossec/etc/certs/root-ca.pem' not found for 'indexer.ssl.certificate_authorities' in module 'indexer'. Check configuration
2025/12/26 11:54:47 wazuh-modulesd: ERROR: (1202): Configuration error at 'etc/ossec.conf'.
wazuh-modulesd: Configuration error. Exiting

 - Configuration finished properly.

 - To start Wazuh:
      /var/ossec/bin/wazuh-control start

 - To stop Wazuh:
      /var/ossec/bin/wazuh-control stop

 - The configuration can be viewed or modified at /var/ossec/etc/ossec.conf


   Thanks for using Wazuh.
   Please don't hesitate to contact us if you need help or find
   any bugs.

   Use our public Mailing List at:
          https://groups.google.com/forum/#!forum/wazuh

   More information can be found at:
          - http://www.wazuh.com

    ---  Press ENTER to finish (maybe more information below). ---


 - In order to connect agent and server, you need to add each agent to the server.

   More information at: 
   https://documentation.wazuh.com/
  • Status
╰─# /var/ossec/bin/wazuh-control  start
2025/12/26 11:59:14 wazuh-modulesd:router: INFO: Loaded router module.
2025/12/26 11:59:14 wazuh-modulesd:content_manager: INFO: Loaded content_manager module.
2025/12/26 11:59:14 wazuh-modulesd:inventory-sync: INFO: Loaded Inventory sync module.
Starting Wazuh v5.0.0...
Started wazuh-apid...
Started wazuh-authd...
Started wazuh-db...
Started wazuh-execd...
Started wazuh-analysisd...
Started wazuh-syscheckd...
Started wazuh-remoted...
Started wazuh-logcollector...
Started wazuh-monitord...
2025/12/26 11:59:17 wazuh-modulesd:router: INFO: Loaded router module.
2025/12/26 11:59:17 wazuh-modulesd:content_manager: INFO: Loaded content_manager module.
2025/12/26 11:59:17 wazuh-modulesd:inventory-sync: INFO: Loaded Inventory sync module.
Started wazuh-modulesd...
Started wazuh-clusterd...
Completed.
  • Built with Tsan
General settings:
    TARGET:             server
    V:                  
    DEBUG:              yes
    INSTALLDIR:         
    DATABASE:           
    ONEWAY:             no
    CLEANFULL:          no
    RESOURCES_URL:      https://packages.wazuh.com/deps/99-29585
    EXTERNAL_SRC_ONLY:  
    HTTP_REQUEST_BRANCH:cd50797cfe03c27f3759bdc243fecca6f7535d35
User settings:
    WAZUH_GROUP:        wazuh
    WAZUH_USER:         wazuh
USE settings:
    USE_INOTIFY:        no
    USE_BIG_ENDIAN:     no
    USE_SELINUX:        no
    USE_AUDIT:          yes
    DISABLE_SYSC:       no
    IMAGE_TRUST_CHECKS: 1
    CA_NAME:            DigiCert Assured ID Root CA
Mysql settings:
    includes:           
    libs:               
Pgsql settings:
    includes:           
    libs:               
Defines:
    -DOSSECHIDS -DUSER="wazuh" -DGROUPGLOBAL="wazuh" -DLinux -DINOTIFY_ENABLED -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DIMAGE_TRUST_CHECKS=1 -DCA_NAME='DigiCert Assured ID Root CA' -DENABLE_SYSC -DENABLE_AUDIT
Compiler:
    CFLAGS            -pthread -Iexternal/pacman/lib/libalpm/ -Iexternal/libarchive/libarchive -Wl,--start-group -Iexternal/audit-userspace/lib -g  -DOSSECHIDS -DUSER="wazuh" -DGROUPGLOBAL="wazuh" -DLinux -DINOTIFY_ENABLED -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DIMAGE_TRUST_CHECKS=1 -DCA_NAME='DigiCert Assured ID Root CA' -DENABLE_SYSC -DENABLE_AUDIT -pipe -Wall -Wextra -std=gnu99 -I./ -I./headers/ -Iexternal/openssl/include -Iexternal/cJSON/ -Iexternal/libyaml/include -Iexternal/curl/include -Iexternal/msgpack/include -Iexternal/bzip2/ -Ishared_modules/common -Ishared_modules/dbsync/include -Ishared_modules/sync_protocol/include -Iwazuh_modules/syscollector/include -Iwazuh_modules/sca/include -Iwazuh_modules/agent_info/include -Idata_provider/include -Iexternal/libpcre2/include -Iexternal/rpm//builddir/output/include -Isyscheckd/include -Ishared_modules/router/include -Ishared_modules/content_manager/include -Ishared_modules/file_helper/file_io/include -Ishared_modules/file_helper/filesystem/include -Iwazuh_modules/vulnerability_scanner/include -I./shared_modules/ -fsanitize=thread -fno-omit-frame-pointer
    LDFLAGS           '-Wl,-rpath,/../lib' -pthread -lrt -ldl  -Lshared_modules/dbsync/build/lib -Lshared_modules/sync_protocol/build/lib -Lshared_modules/file_helper/build/lib -fsanitize=thread -Lwazuh_modules/syscollector/build/lib -Lwazuh_modules/sca/build/lib -Lwazuh_modules/agent_info/build/lib -Ldata_provider/build/lib -Lsyscheckd/build/lib
    LIBS              -lrt -ldl -lm 
    CC                gcc
    MAKE              /usr/bin/make
make[1]: Leaving directory '/workspaces/wazuh-5.x-v1/wazuh/src'

Done building server
  • Check

After replacing the binary and executing the engine-test in several consoles having this report

ThreadSanitizer.txt

extracting an example:

WARNING: ThreadSanitizer: data race (pid=1521585)
  Write of size 8 at 0x72c426cb4000 by thread T243 (mutexes: read M0):
    #0 free <null> (libtsan.so.2+0x52bac)
    #1 rapidjson::CrtAllocator::Free(void*) /workspaces/wazuh-5.x-v1/wazuh/src/external/rapidjson/include/rapidjson/allocators.h:100 (wazuh-analysisdTsan+0x11c9f1e)

  Previous write of size 8 at 0x72c426cb4000 by thread T262:
    #0 malloc <null> (libtsan.so.2+0x502e2)
    #1 rapidjson::CrtAllocator::Malloc(unsigned long) /workspaces/wazuh-5.x-v1/wazuh/src/external/rapidjson/include/rapidjson/allocators.h:88 (wazuh-analysisdTsan+0x12aaa8a)

  Mutex M0 (0x722c00000d78) created at:
    #0 pthread_rwlock_wrlock <null> (libtsan.so.2+0x52708)
    #1 __glibcxx_rwlock_wrlock /opt/gcc-14/include/c++/14.3.0/shared_mutex:85 (wazuh-analysisdTsan+0x11d3048)

  Thread T243 'ORWorker-116963' (tid=1522124, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.2+0x555e2)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (wazuh-analysisdTsan+0x1a61f08)

  Thread T262 (tid=1522143, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.2+0x555e2)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (wazuh-analysisdTsan+0x1a61f08)

SUMMARY: ThreadSanitizer: data race (/opt/gcc-14/lib64/libtsan.so.2+0x52bac) in free

besides that several checks didn't passed as succeeded

@jam300 jam300 force-pushed the bug/32387-Analyze_TSAN branch 3 times, most recently from 28f13db to 2c51ac8 Compare December 27, 2025 05:16
@jam300 jam300 requested a review from LucioDonda December 29, 2025 14:30
LucioDonda
LucioDonda approved these changes Dec 29, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@LucioDonda LucioDonda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

  • Rebuilt and replacing with tsan
╰─# TSAN_OPTIONS="log_path=/tmp/tsan_report_check:halt_on_error=0:abort_on_error=0:exitcode=0:print_stacktrace=1:detect_deadlocks=0:detect_thread_leaks=1" \
WAZUH_ORCHESTRATOR_THREADS=2 \
LD_LIBRARY_PATH=/workspaces/wazuh-5.x-v1/wazuh/src/build/lib:/workspaces/wazuh-5.x-v1/wazuh/src/build/engine:/var/ossec/lib \
/var/ossec/bin/wazuh-analysisdTsan -f
2025/12/29 18:37:07 wazuh-analysisd: INFO: Store initialized.
2025/12/29 18:37:07 wazuh-analysisd: INFO: Initializing CTI Store ContentManager
2025/12/29 18:37:07 wazuh-analysisd: INFO: ContentManager: CTIStorageDB opened at '/var/ossec/engine/cti/assets_database' (open=true)
2025/12/29 18:37:07 wazuh-analysisd: INFO: CTI Store initialized
2025/12/29 18:37:07 wazuh-analysisd: INFO: Starting CTI Store content synchronization
2025/12/29 18:37:07 wazuh-analysisd: INFO: Starting CTI Store ContentDownloader
2025/12/29 18:37:07 wazuh-analysisd: INFO: OAuth authentication enabled for CTI Store (productType: catalog:consumer:decoders)
2025/12/29 18:37:07 wazuh-analysisd: INFO: CTI Store ContentDownloader started successfully
2025/12/29 18:37:07 wazuh-analysisd: INFO: Content Manager initialized.
2025/12/29 18:37:07 wazuh-analysisd: INFO: KVDB initialized.
2025/12/29 18:37:07 wazuh-analysisd: INFO: Geo initialized.
2025/12/29 18:37:12 wazuh-analysisd: INFO: Schema initialized.
2025/12/29 18:37:13 wazuh-modulesd:content-updater: WARNING: CtiDownloader: Failed to fetch credentials from Indexer after 3 attempts. Falling back to default URL
2025/12/29 18:37:13 wazuh-analysisd: INFO: Loaded timezone database version: '2025c'
2025/12/29 18:37:13 wazuh-analysisd: INFO: HLP initialized.
2025/12/29 18:37:13 IndexerConnector: WARNING: No username and password found in the keystore, using default values.
2025/12/29 18:37:13 wazuh-analysisd: INFO: Indexer Connector initialized.
2025/12/29 18:37:13 wazuh-analysisd: INFO: Scheduler initialized and started.
2025/12/29 18:37:13 wazuh-analysisd: INFO: Stream logger initialized.
2025/12/29 18:37:13 wazuh-analysisd: INFO: No flooding file provided, the queue will not be flooded.
2025/12/29 18:37:13 wazuh-analysisd: INFO: No flooding file provided, the queue will not be flooded.
2025/12/29 18:37:13 wazuh-analysisd: INFO: Builder initialized.
2025/12/29 18:37:13 wazuh-analysisd: INFO: Content Manager CRUD Service initialized.
2025/12/29 18:37:13 wazuh-analysisd: INFO: No flooding file provided, the queue will not be flooded.
2025/12/29 18:37:13 wazuh-analysisd: INFO: No flooding file provided, the queue will not be flooded.
2025/12/29 18:37:13 wazuh-analysisd: WARNING: Router: router/tester/0 table is empty
2025/12/29 18:37:58 wazuh-analysisd: INFO: Router initialized.
2025/12/29 18:37:58 wazuh-analysisd: INFO: Archiver initialized.
2025/12/29 18:37:58 wazuh-analysisd: INFO: Server API_SRV started in thread 140311643805248 at /var/ossec/queue/sockets/analysis
2025/12/29 18:37:58 wazuh-analysisd: INFO: Local engine's server initialized and started.
2025/12/29 18:37:58 wazuh-analysisd: INFO: Server ENRICHED_EVENTS_SRV started in thread 140311496988224 at /var/ossec/queue/sockets/queue-http.sock
2025/12/29 18:37:58 wazuh-analysisd: INFO: Remote engine's server initialized and started.
  • Test executed:

╰─# time engine-test run test -n cti -dd 

.result.columns.size' is not a string or it doesn't exist
  ↳ osquery.result.columns.start_interval: parse_long($osquery.result.columns.start_interval) -> Reference 'osquery.result.columns.start_interval' is not a string or it doesn't exist
  ↳ osquery.result.columns.start_on_mount: get_key_in({"0":false,"1":true}, $osquery.result.columns.start_on_mount) -> Reference 'osquery.result.columns.start_on_mount' not found
  ↳ osquery.result.columns.start_time: parse_long($osquery.result.columns.start_time) -> Reference 'osquery.result.columns.start_time' is not a string or it doesn't exist
  ↳ osquery.result.columns.start_time: date_from_epoch($osquery.result.columns.start_time) -> Reference 'osquery.result.columns.start_time' not found
  ↳ osquery.result.columns.state: parse_long($osquery.result.columns.state) -> Reference 'osquery.result.columns.state' is not a string or it doesn't exist
  ↳ osquery.result.columns.stealth_enabled: get_key_in({"0":false,"1":true}, $osquery.result.columns.stealth_enabled) -> Reference 'osquery.result.columns.stealth_enabled' not found
  ↳ osquery.result.columns.symlink: get_key_in({"0":false,"1":true}, $osquery.result.columns.symlink) -> Reference 'osquery.result.columns.symlink' not found
  ↳ osquery.result.columns.timestamp: parse_date($osquery.result.columns.timestamp, "%a %b %d %H:%M:%S %Y %Z") -> Reference 'osquery.result.columns.timestamp' is not a string or it doesn't exist
  ↳ osquery.result.columns.unix_time: parse_long($osquery.result.columns.unix_time) -> Reference 'osquery.result.columns.unix_time' is not a string or it doesn't exist
  ↳ osquery.result.columns.unix_time: date_from_epoch($osquery.result.columns.unix_time) -> Reference 'osquery.result.columns.unix_time' not found
  ↳ osquery.result.columns.usb_address: parse_long($osquery.result.columns.usb_address) -> Reference 'osquery.result.columns.usb_address' is not a string or it doesn't exist
  ↳ osquery.result.columns.usb_port: parse_long($osquery.result.columns.usb_port) -> Reference 'osquery.result.columns.usb_port' is not a string or it doesn't exist
  ↳ osquery.result.columns.visible: get_key_in({"0":false,"1":true}, $osquery.result.columns.visible) -> Reference 'osquery.result.columns.visible' not found
  ↳ osquery.result.columns.wall_time: parse_long($osquery.result.columns.wall_time) -> Success
  ↳ osquery.result.columns.watcher: parse_long($osquery.result.columns.watcher) -> Reference 'osquery.result.columns.watcher' is not a string or it doesn't exist
  ↳ osquery.result.columns.year: parse_long($osquery.result.columns.year) -> Reference 'osquery.result.columns.year' is not a string or it doesn't exist
  ↳ osquery.result.counter: parse_long($osquery.result.counter) -> Success
  ↳ osquery.result.epoch: parse_long($osquery.result.epoch) -> Success
  ↳ osquery.result.epoch: date_from_epoch($osquery.result.epoch) -> Success
  ↳ osquery.result.unix_time: parse_long($osquery.result.unix_time) -> Success
  ↳ osquery.result.unix_time: date_from_epoch($osquery.result.unix_time) -> Success

output:
  '@timestamp': '2025-12-29T18:46:47Z'
  agent:
    id: '000'
    name: 09c2e042038b
  event:
    action: added
    kind: event
    original: '{"action":"added","calendarTime":"Mon Jan  8 14:51:55 2018 UTC","columns":{"average_memory":"0","avg_system_time":"0","avg_user_time":"0","executions":"40","interval":"60","last_executed":"1515423113","name":"pack_ossec-rootkit_ark_rootkit","output_size":"0","wall_time":"0"},"counter":"0","decorations":{"host_uuid":"72E1287B-D1BC-4FC6-B9D8-64F4352776A9","username":"ubuntu"},"epoch":"0","hostIdentifier":"ubuntu-xenial","name":"pack_osquery-monitoring_schedule","unixTime":"1515423115"}'
    type:
    - info
  host:
    architecture: x86_64
    hostname: ubuntu-xenial
    id: 72E1287B-D1BC-4FC6-B9D8-64F4352776A9
    ip:
    - 172.17.0.2
    os:
      full: jammy
      kernel: Linux |09c2e042038b |5.15.0-144-generic |#157-Ubuntu SMP Mon Jun 16
        07:33:10 UTC 2025 |x86_64
      name: Ubuntu
      platform: ubuntu
      version: 22.04.5 LTS (Jammy Jellyfish)
  osquery:
    result:
      action: added
      calendar_time: '2018-01-08T14:51:55.000Z'
      columns:
        average_memory: 0
        avg_system_time: 0
        avg_user_time: 0
        executions: 40
        interval: 60
        last_executed: '2018-01-08T14:51:53.000000Z'
        name: pack_ossec-rootkit_ark_rootkit
        output_size: 0
        wall_time: 0
      counter: 0
      decorations:
        host_uuid: 72E1287B-D1BC-4FC6-B9D8-64F4352776A9
        username: ubuntu
      epoch: '1970-01-01T00:00:00.000000Z'
      host_identifier: ubuntu-xenial
      name: pack_osquery-monitoring_schedule
      unix_time: '2018-01-08T14:51:55.000000Z'
  related:
    hosts:
    - ubuntu-xenial
    user:
    - ubuntu
  rule:
    name: pack_osquery-monitoring_schedule
  user:
    name: ubuntu
  wazuh:
    integration:
      category: security
      decoders:
      - core-wazuh-message
      - integrations
      - osquery-default
      name: osquery
    protocol:
      location: some
      queue: 49



Enter any events [ENTER to send event, CTRL+C to finish]:
  • Check report presence:
╰─# ls -l /tmp/tsan_report_check
ls: cannot access '/tmp/tsan_report_check': No such file or directory

@jam300 jam300 force-pushed the bug/32387-Analyze_TSAN branch from 2c51ac8 to b146332 Compare December 30, 2025 03:34
NahuFigueroa97
NahuFigueroa97 reviewed Dec 30, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@NahuFigueroa97 NahuFigueroa97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I ran engine-test in the cti namespace and inserted events, I got this report in TSAN

2025/12/30 14:58:59 wazuh-analysisd: INFO: Engine started and ready to process events.
==================
WARNING: ThreadSanitizer: data race (pid=4984)
  Read of size 8 at 0x72080167a6a0 by thread T202 (mutexes: read M0):
    #0 memcmp <null> (libtsan.so.2+0x583f0)
    #1 std::char_traits<char>::compare(char const*, char const*, unsigned long) /opt/gcc-14/include/c++/14.3.0/bits/char_traits.h:381 (wazuh-analysisd+0x11cffed)

  Previous write of size 8 at 0x72080167a6a0 by thread T205:
    #0 memcpy <null> (libtsan.so.2+0x804b8)
    #1 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) /opt/gcc-14/include/c++/14.3.0/bits/basic_string.tcc:247 (wazuh-analysisd+0x120bc83)

  As if synchronized via sleep:
    #0 nanosleep <null> (libtsan.so.2+0x539ab)
    #1 void std::this_thread::sleep_for<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&) /opt/gcc-14/include/c++/14.3.0/bits/this_thread_sleep.h:80 (wazuh-analysisd+0x11f9f6d)

  Location is heap block of size 18 at 0x72080167a6a0 allocated by thread T205:
    #0 operator new(unsigned long) <null> (libtsan.so.2+0x99ec2)
    #1 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) /opt/gcc-14/include/c++/14.3.0/bits/basic_string.tcc:229 (wazuh-analysisd+0x120bc1e)

  Mutex M0 (0x7228000015a0) created at:
    #0 pthread_rwlock_wrlock <null> (libtsan.so.2+0x52708)
    #1 __glibcxx_rwlock_wrlock /opt/gcc-14/include/c++/14.3.0/shared_mutex:85 (wazuh-analysisd+0x11ee385)

  Thread T202 'ORWorker-861835' (tid=5454, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.2+0x555e2)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (wazuh-analysisd+0x1a925c8)

  Thread T205 (tid=5457, running) created by thread T203 at:
    #0 pthread_create <null> (libtsan.so.2+0x555e2)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (wazuh-analysisd+0x1a925c8)

SUMMARY: ThreadSanitizer: data race (/opt/gcc-14/lib64/libtsan.so.2+0x583f0) in memcmp
==================
==================
WARNING: ThreadSanitizer: data race (pid=4984)
  Read of size 8 at 0x7210025bfd00 by thread T202 (mutexes: read M0):
    #0 memcpy <null> (libtsan.so.2+0x804b8)
    #1 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) /opt/gcc-14/include/c++/14.3.0/bits/basic_string.tcc:247 (wazuh-analysisd+0x120bc83)

  Previous write of size 8 at 0x7210025bfd00 by thread T205:
    #0 memcpy <null> (libtsan.so.2+0x804b8)
    #1 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) /opt/gcc-14/include/c++/14.3.0/bits/basic_string.tcc:247 (wazuh-analysisd+0x120bc83)

  As if synchronized via sleep:
    #0 nanosleep <null> (libtsan.so.2+0x539ab)
    #1 void std::this_thread::sleep_for<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&) /opt/gcc-14/include/c++/14.3.0/bits/this_thread_sleep.h:80 (wazuh-analysisd+0x11f9f6d)

  Location is heap block of size 51 at 0x7210025bfd00 allocated by thread T205:
    #0 operator new(unsigned long) <null> (libtsan.so.2+0x99ec2)
    #1 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) /opt/gcc-14/include/c++/14.3.0/bits/basic_string.tcc:229 (wazuh-analysisd+0x120bc1e)

  Mutex M0 (0x7228000015a0) created at:
    #0 pthread_rwlock_wrlock <null> (libtsan.so.2+0x52708)
    #1 __glibcxx_rwlock_wrlock /opt/gcc-14/include/c++/14.3.0/shared_mutex:85 (wazuh-analysisd+0x11ee385)

  Thread T202 'ORWorker-861835' (tid=5454, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.2+0x555e2)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (wazuh-analysisd+0x1a925c8)

  Thread T205 (tid=5457, running) created by thread T203 at:
    #0 pthread_create <null> (libtsan.so.2+0x555e2)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (wazuh-analysisd+0x1a925c8)

SUMMARY: ThreadSanitizer: data race (/opt/gcc-14/lib64/libtsan.so.2+0x804b8) in memcpy
==================
==================
WARNING: ThreadSanitizer: data race (pid=4984)
  Read of size 5 at 0x72c426d7c018 by thread T202 (mutexes: read M0):
    #0 memcmp <null> (libtsan.so.2+0x583f0)
    #1 bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::StringEqual<rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&) const /workspaces/devContainer/wazuh/src/external/rapidjson/include/rapidjson/document.h:2478 (wazuh-analysisd+0x12cdb1f)

  Previous write of size 8 at 0x72c426d7c018 by thread T205:
    #0 malloc <null> (libtsan.so.2+0x502e2)
    #1 rapidjson::CrtAllocator::Malloc(unsigned long) /workspaces/devContainer/wazuh/src/external/rapidjson/include/rapidjson/allocators.h:88 (wazuh-analysisd+0x12c6688)

  As if synchronized via sleep:
    #0 nanosleep <null> (libtsan.so.2+0x539ab)
    #1 void std::this_thread::sleep_for<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&) /opt/gcc-14/include/c++/14.3.0/bits/this_thread_sleep.h:80 (wazuh-analysisd+0x11f9f6d)

  Location is heap block of size 65560 at 0x72c426d7c000 allocated by thread T205:
    #0 malloc <null> (libtsan.so.2+0x502e2)
    #1 rapidjson::CrtAllocator::Malloc(unsigned long) /workspaces/devContainer/wazuh/src/external/rapidjson/include/rapidjson/allocators.h:88 (wazuh-analysisd+0x12c6688)

  Mutex M0 (0x7228000015a0) created at:
    #0 pthread_rwlock_wrlock <null> (libtsan.so.2+0x52708)
    #1 __glibcxx_rwlock_wrlock /opt/gcc-14/include/c++/14.3.0/shared_mutex:85 (wazuh-analysisd+0x11ee385)

  Thread T202 'ORWorker-861835' (tid=5454, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.2+0x555e2)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (wazuh-analysisd+0x1a925c8)

  Thread T205 (tid=5457, running) created by thread T203 at:
    #0 pthread_create <null> (libtsan.so.2+0x555e2)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (wazuh-analysisd+0x1a925c8)

SUMMARY: ThreadSanitizer: data race (/opt/gcc-14/lib64/libtsan.so.2+0x583f0) in memcmp
==================
==================
WARNING: ThreadSanitizer: data race (pid=4984)
  Read of size 6 at 0x72c426d7c038 by thread T202 (mutexes: read M0):
    #0 strlen <null> (libtsan.so.2+0x5672c)
    #1 std::char_traits<char>::length(char const*) /opt/gcc-14/include/c++/14.3.0/bits/char_traits.h:391 (wazuh-analysisd+0x11d001f)

  Previous write of size 8 at 0x72c426d7c038 by thread T205:
    #0 malloc <null> (libtsan.so.2+0x502e2)
    #1 rapidjson::CrtAllocator::Malloc(unsigned long) /workspaces/devContainer/wazuh/src/external/rapidjson/include/rapidjson/allocators.h:88 (wazuh-analysisd+0x12c6688)

  As if synchronized via sleep:
    #0 nanosleep <null> (libtsan.so.2+0x539ab)
    #1 void std::this_thread::sleep_for<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&) /opt/gcc-14/include/c++/14.3.0/bits/this_thread_sleep.h:80 (wazuh-analysisd+0x11f9f6d)

  Location is heap block of size 65560 at 0x72c426d7c000 allocated by thread T205:
    #0 malloc <null> (libtsan.so.2+0x502e2)
    #1 rapidjson::CrtAllocator::Malloc(unsigned long) /workspaces/devContainer/wazuh/src/external/rapidjson/include/rapidjson/allocators.h:88 (wazuh-analysisd+0x12c6688)

  Mutex M0 (0x7228000015a0) created at:
    #0 pthread_rwlock_wrlock <null> (libtsan.so.2+0x52708)
    #1 __glibcxx_rwlock_wrlock /opt/gcc-14/include/c++/14.3.0/shared_mutex:85 (wazuh-analysisd+0x11ee385)

  Thread T202 'ORWorker-861835' (tid=5454, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.2+0x555e2)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (wazuh-analysisd+0x1a925c8)

  Thread T205 (tid=5457, running) created by thread T203 at:
    #0 pthread_create <null> (libtsan.so.2+0x555e2)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (wazuh-analysisd+0x1a925c8)

SUMMARY: ThreadSanitizer: data race (/opt/gcc-14/lib64/libtsan.so.2+0x5672c) in strlen
==================
==================
WARNING: ThreadSanitizer: data race (pid=4984)
  Read of size 8 at 0x72c426d7c7f8 by thread T202 (mutexes: read M0):
    #0 strlen <null> (libtsan.so.2+0x5672c)
    #1 std::char_traits<char>::length(char const*) /opt/gcc-14/include/c++/14.3.0/bits/char_traits.h:391 (wazuh-analysisd+0x11d001f)

  Previous write of size 8 at 0x72c426d7c7f8 by thread T205:
    #0 memcpy <null> (libtsan.so.2+0x804b8)
    #1 rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::SetStringRaw(rapidjson::GenericStringRef<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>&) /workspaces/devContainer/wazuh/src/external/rapidjson/include/rapidjson/document.h:2454 (wazuh-analysisd+0x12ce24a)

  As if synchronized via sleep:
    #0 nanosleep <null> (libtsan.so.2+0x539ab)
    #1 void std::this_thread::sleep_for<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&) /opt/gcc-14/include/c++/14.3.0/bits/this_thread_sleep.h:80 (wazuh-analysisd+0x11f9f6d)

  Location is heap block of size 65560 at 0x72c426d7c000 allocated by thread T205:
    #0 malloc <null> (libtsan.so.2+0x502e2)
    #1 rapidjson::CrtAllocator::Malloc(unsigned long) /workspaces/devContainer/wazuh/src/external/rapidjson/include/rapidjson/allocators.h:88 (wazuh-analysisd+0x12c6688)

  Mutex M0 (0x7228000015a0) created at:
    #0 pthread_rwlock_wrlock <null> (libtsan.so.2+0x52708)
    #1 __glibcxx_rwlock_wrlock /opt/gcc-14/include/c++/14.3.0/shared_mutex:85 (wazuh-analysisd+0x11ee385)

  Thread T202 'ORWorker-861835' (tid=5454, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.2+0x555e2)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (wazuh-analysisd+0x1a925c8)

  Thread T205 (tid=5457, running) created by thread T203 at:
    #0 pthread_create <null> (libtsan.so.2+0x555e2)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (wazuh-analysisd+0x1a925c8)

SUMMARY: ThreadSanitizer: data race (/opt/gcc-14/lib64/libtsan.so.2+0x5672c) in strlen
==================
==================
WARNING: ThreadSanitizer: data race (pid=4984)
  Read of size 8 at 0x72c426d7c658 by thread T202 (mutexes: read M0):
    #0 memcpy <null> (libtsan.so.2+0x804b8)
    #1 rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>::Realloc(void*, unsigned long, unsigned long) /workspaces/devContainer/wazuh/src/external/rapidjson/include/rapidjson/allocators.h:364 (wazuh-analysisd+0x12d8d3d)

  Previous write of size 8 at 0x72c426d7c658 by thread T205:
    #0 malloc <null> (libtsan.so.2+0x502e2)
    #1 rapidjson::CrtAllocator::Malloc(unsigned long) /workspaces/devContainer/wazuh/src/external/rapidjson/include/rapidjson/allocators.h:88 (wazuh-analysisd+0x12c6688)

  As if synchronized via sleep:
    #0 nanosleep <null> (libtsan.so.2+0x539ab)
    #1 void std::this_thread::sleep_for<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&) /opt/gcc-14/include/c++/14.3.0/bits/this_thread_sleep.h:80 (wazuh-analysisd+0x11f9f6d)

  Location is heap block of size 65560 at 0x72c426d7c000 allocated by thread T205:
    #0 malloc <null> (libtsan.so.2+0x502e2)
    #1 rapidjson::CrtAllocator::Malloc(unsigned long) /workspaces/devContainer/wazuh/src/external/rapidjson/include/rapidjson/allocators.h:88 (wazuh-analysisd+0x12c6688)

  Mutex M0 (0x7228000015a0) created at:
    #0 pthread_rwlock_wrlock <null> (libtsan.so.2+0x52708)
    #1 __glibcxx_rwlock_wrlock /opt/gcc-14/include/c++/14.3.0/shared_mutex:85 (wazuh-analysisd+0x11ee385)

  Thread T202 'ORWorker-861835' (tid=5454, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.2+0x555e2)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (wazuh-analysisd+0x1a925c8)

  Thread T205 (tid=5457, running) created by thread T203 at:
    #0 pthread_create <null> (libtsan.so.2+0x555e2)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (wazuh-analysisd+0x1a925c8)

SUMMARY: ThreadSanitizer: data race (/opt/gcc-14/lib64/libtsan.so.2+0x804b8) in memcpy
==================

NahuFigueroa97
NahuFigueroa97 approved these changes Dec 30, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@NahuFigueroa97 NahuFigueroa97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

matigarciadev
matigarciadev approved these changes Dec 30, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@matigarciadev matigarciadev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Built and Install:

General settings:
    TARGET:             server
    V:                  
    DEBUG:              
    INSTALLDIR:         /var/ossec
    DATABASE:           
    ONEWAY:             no
    CLEANFULL:          no
    RESOURCES_URL:      https://packages.wazuh.com/deps/99-29585
    EXTERNAL_SRC_ONLY:  
    HTTP_REQUEST_BRANCH:cd50797cfe03c27f3759bdc243fecca6f7535d35
User settings:
    WAZUH_GROUP:        wazuh
    WAZUH_USER:         wazuh
USE settings:
    USE_INOTIFY:        no
    USE_BIG_ENDIAN:     no
    USE_SELINUX:        no
    USE_AUDIT:          yes
    DISABLE_SYSC:       no
    IMAGE_TRUST_CHECKS: 1
    CA_NAME:            DigiCert Assured ID Root CA
Mysql settings:
    includes:           
    libs:               
Pgsql settings:
    includes:           
    libs:               
Defines:
    -DOSSECHIDS -DUSER="wazuh" -DGROUPGLOBAL="wazuh" -DLinux -DINOTIFY_ENABLED -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DIMAGE_TRUST_CHECKS=1 -DCA_NAME='DigiCert Assured ID Root CA' -DENABLE_SYSC -DENABLE_AUDIT
Compiler:
    CFLAGS            -pthread -Iexternal/pacman/lib/libalpm/ -Iexternal/libarchive/libarchive -Wl,--start-group -Iexternal/audit-userspace/lib -g -DNDEBUG -O2 -DOSSECHIDS -DUSER="wazuh" -DGROUPGLOBAL="wazuh" -DLinux -DINOTIFY_ENABLED -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DIMAGE_TRUST_CHECKS=1 -DCA_NAME='DigiCert Assured ID Root CA' -DENABLE_SYSC -DENABLE_AUDIT -pipe -Wall -Wextra -std=gnu99 -I./ -I./headers/ -Iexternal/openssl/include -Iexternal/cJSON/ -Iexternal/libyaml/include -Iexternal/curl/include -Iexternal/msgpack/include -Iexternal/bzip2/ -Ishared_modules/common -Ishared_modules/dbsync/include -Ishared_modules/sync_protocol/include -Iwazuh_modules/syscollector/include -Iwazuh_modules/sca/include -Iwazuh_modules/agent_info/include -Idata_provider/include -Iexternal/libpcre2/include -Iexternal/rpm//builddir/output/include -Isyscheckd/include -Ishared_modules/router/include -Ishared_modules/content_manager/include -Ishared_modules/file_helper/file_io/include -Ishared_modules/file_helper/filesystem/include -Iwazuh_modules/vulnerability_scanner/include -I./shared_modules/ 
    LDFLAGS           '-Wl,-rpath,/../lib' -pthread -lrt -ldl -O2 -Lshared_modules/dbsync/build/lib -Lshared_modules/sync_protocol/build/lib -Lshared_modules/file_helper/build/lib  -Lwazuh_modules/syscollector/build/lib -Lwazuh_modules/sca/build/lib -Lwazuh_modules/agent_info/build/lib -Ldata_provider/build/lib -Lsyscheckd/build/lib
    LIBS              -lrt -ldl -lm 
    CC                gcc
    MAKE              make
make[1]: Leaving directory '/workspaces/wazuh-5.x/wazuh/src'

Done building server
...
Completed.

 - Configuration finished properly.

 - To start Wazuh:
      /var/ossec/bin/wazuh-control start

 - To stop Wazuh:
      /var/ossec/bin/wazuh-control stop

 - The configuration can be viewed or modified at /var/ossec/etc/ossec.conf


   Thanks for using Wazuh.
   Please don't hesitate to contact us if you need help or find
   any bugs.

   Use our public Mailing List at:
          https://groups.google.com/forum/#!forum/wazuh

   More information can be found at:
          - http://www.wazuh.com

    ---  Press ENTER to finish (maybe more information below). ---

 - Update completed.

Unit Tests compiled with TSAN

 ctest --test-dir build/engine --output-on-failure -j "$(nproc)"
...
2 00-00 C0-FB 01-00 08-72 00-00 EB-07 00-00 00-00 00-00 B0-02 04-00 20-72 00-00 03-00 00-00 00-00 00-00 61-6E 64-00 00-00 00-00 00-00... (Skipped)
        782 - BK/PipelineTest.RxProcessTraces/("Complex: And of broadcast", (ptr = 0x72200003ff00, value = 96-byte object <58-E4 BC-00 00-00 00-00 00-FF 03-00 20-72 00-00 E0-FD 01-00 08-72 00-00 E4-07 00-00 00-00 00-00 30-FF 03-00 20-72 00-00 03-00 00-00 00-00 00-00 61-6E 64-00 00-00 00-00 00-00... (Skipped)
        783 - BK/PipelineTest.RxProcessTraces/("Complex: And of chain", (ptr = 0x72200003fb80, value = 96-byte object <58-E4 BC-00 00-00 00-00 80-FB 03-00 20-72 00-00 00-00 02-00 08-72 00-00 DD-07 00-00 00-00 00-00 B0-FB 03-00 20-72 00-00 03-00 00-00 00-00 00-00 61-6E 64-00 00-00 00-00 00-00 00-... (Skipped)
        784 - BK/PipelineTest.RxProcessTraces/("Complex: And of chain", (ptr = 0x72200003f800, value = 96-byte object <58-E4 BC-00 00-00 00-00 00-F8 03-00 20-72 00-00 20-39 01-00 08-72 00-00 D6-07 00-00 00-00 00-00 30-F8 03-00 20-72 00-00 03-00 00-00 00-00 00-00 61-6E 64-00 00-00 00-00 00-00 00-... (Skipped)
        785 - BK/PipelineTest.RxProcessTraces/("Complex: And of implication", (ptr = 0x72200003f480, value = 96-byte object <58-E4 BC-00 00-00 00-00 80-F4 03-00 20-72 00-00 20-7A 01-00 08-72 00-00 CF-07 00-00 00-00 00-00 B0-F4 03-00 20-72 00-00 03-00 00-00 00-00 00-00 61-6E 64-00 00-00 00-00 00-... (Skipped)
        786 - BK/PipelineTest.RxProcessTraces/("Complex: And of implication", (ptr = 0x72200003f100, value = 96-byte object <58-E4 BC-00 00-00 00-00 00-F1 03-00 20-72 00-00 00-95 01-00 08-72 00-00 C8-07 00-00 00-00 00-00 30-F1 03-00 20-72 00-00 03-00 00-00 00-00 00-00 61-6E 64-00 00-00 00-00 00-... (Skipped)
        787 - BK/PipelineTest.RxProcessTraces/("Complex: And of implication", (ptr = 0x72200003ed80, value = 96-byte object <58-E4 BC-00 00-00 00-00 80-ED 03-00 20-72 00-00 E0-B7 01-00 08-72 00-00 C1-07 00-00 00-00 00-00 B0-ED 03-00 20-72 00-00 03-00 00-00 00-00 00-00 61-6E 64-00 00-00 00-00 00-... (Skipped)
        788 - BK/PipelineTest.RxProcessTraces/("Complex: And of implication", (ptr = 0x72200003ea00, value = 96-byte object <58-E4 BC-00 00-00 00-00 00-EA 03-00 20-72 00-00 60-DB 01-00 08-72 00-00 BA-07 00-00 00-00 00-00 30-EA 03-00 20-72 00-00 03-00 00-00 00-00 00-00 61-6E 64-00 00-00 00-00 00-... (Skipped)
        789 - BK/PipelineTest.RxProcessTraces/("Complex: And of implication", (ptr = 0x72200003e680, value = 96-byte object <58-E4 BC-00 00-00 00-00 80-E6 03-00 20-72 00-00 A0-DE 01-00 08-72 00-00 B3-07 00-00 00-00 00-00 B0-E6 03-00 20-72 00-00 03-00 00-00 00-00 00-00 61-6E 64-00 00-00 00-00 00-... (Skipped)
        790 - BK/PipelineTest.RxProcessTraces/("Complex: And of or", (ptr = 0x72200003e300, value = 96-byte object <58-E4 BC-00 00-00 00-00 00-E3 03-00 20-72 00-00 E0-D4 01-00 08-72 00-00 AC-07 00-00 00-00 00-00 30-E3 03-00 20-72 00-00 03-00 00-00 00-00 00-00 61-6E 64-00 00-00 00-00 00-00 00-00 ... (Skipped)
        791 - BK/PipelineTest.RxProcessTraces/("Complex: And of or", (ptr = 0x72200003df80, value = 96-byte object <58-E4 BC-00 00-00 00-00 80-DF 03-00 20-72 00-00 E0-EE 01-00 08-72 00-00 A5-07 00-00 00-00 00-00 B0-DF 03-00 20-72 00-00 03-00 00-00 00-00 00-00 61-6E 64-00 00-00 00-00 00-00 00-00 ... (Skipped)
        792 - BK/PipelineTest.RxProcessTraces/("Complex: And of or", (ptr = 0x72200003dc00, value = 96-byte object <58-E4 BC-00 00-00 00-00 00-DC 03-00 20-72 00-00 C0-F0 01-00 08-72 00-00 9E-07 00-00 00-00 00-00 30-DC 03-00 20-72 00-00 03-00 00-00 00-00 00-00 61-6E 64-00 00-00 00-00 00-00 00-00 ... (Skipped)
        793 - BK/PipelineTest.RxProcessTraces/("Complex: And of and", (ptr = 0x72200003d880, value = 96-byte object <58-E4 BC-00 00-00 00-00 80-D8 03-00 20-72 00-00 E0-F2 01-00 08-72 00-00 97-07 00-00 00-00 00-00 B0-D8 03-00 20-72 00-00 03-00 00-00 00-00 00-00 61-6E 64-00 00-00 00-00 00-00 00-00... (Skipped)
        794 - BK/PipelineTest.RxProcessTraces/("Complex: And of and", (ptr = 0x72200003d500, value = 96-byte object <58-E4 BC-00 00-00 00-00 00-D5 03-00 20-72 00-00 A0-F4 01-00 08-72 00-00 90-07 00-00 00-00 00-00 30-D5 03-00 20-72 00-00 03-00 00-00 00-00 00-00 61-6E 64-00 00-00 00-00 00-00 00-00... (Skipped)
        795 - BK/PipelineTest.RxProcessTraces/("Complex: And of and", (ptr = 0x72200003d180, value = 96-byte object <58-E4 BC-00 00-00 00-00 80-D1 03-00 20-72 00-00 A0-F6 01-00 08-72 00-00 89-07 00-00 00-00 00-00 B0-D1 03-00 20-72 00-00 03-00 00-00 00-00 00-00 61-6E 64-00 00-00 00-00 00-00 00-00... (Skipped)
        6880 - ChannelHandlerTest.ErrorHandlingNonWritableDirectory (Skipped)
        6934 - LoggerTest.LoggerInitialization (Skipped)
        7073 - WIndexerConnectorTest.IntegrationTest (Disabled)
        7202 - ContentManagerTest.init (Skipped)
        7258 - ContentManagerTest.init (Skipped)

echo $?                                                         
0

Result: No TSAN reports were observed in the unit tests.

@jam300 jam300 force-pushed the bug/32387-Analyze_TSAN branch from b146332 to 766c444 Compare December 30, 2025 19:25
juliancnn
juliancnn approved these changes Dec 30, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@juliancnn juliancnn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@jam300 jam300 force-pushed the bug/32387-Analyze_TSAN branch from 766c444 to 1bb56fa Compare December 30, 2025 20:52
@jam300 jam300 force-pushed the bug/32387-Analyze_TSAN branch from 1bb56fa to 9823cf0 Compare December 30, 2025 22:19
@juliancnn juliancnn merged commit 80445e8 into main Dec 31, 2025
29 checks passed
@juliancnn juliancnn deleted the bug/32387-Analyze_TSAN branch December 31, 2025 01:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@juliancnn juliancnn juliancnn approved these changes

@NahuFigueroa97 NahuFigueroa97 NahuFigueroa97 approved these changes

@matigarciadev matigarciadev matigarciadev approved these changes

@LucioDonda LucioDonda LucioDonda approved these changes

Assignees

@jam300 jam300

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Spike: Analyze TSAN-report of wazuh-engine in test mode

5 participants

@jam300 @matigarciadev @juliancnn @NahuFigueroa97 @LucioDonda