Skip to content

Silence untrustworthy FIM schema-validation errors during shutdown - #37688

Merged
Nicogp merged 1 commit into
5.0.0from
fix/37654-schema-validation-log-on-shutdown
Jul 14, 2026
Merged

Nicogp merged 1 commit into
5.0.0from
fix/37654-schema-validation-log-on-shutdown

Conversation

@Nicogp

@Nicogp Nicogp commented Jul 14, 2026

Copy link
Copy Markdown
Member

Description

On agent shutdown, wazuh-syscheckd can emit a misleading ERROR reporting that a valid event failed schema validation because of an "invalid" ISO8601 date that is in fact perfectly valid:

wazuh-syscheckd: ERROR: Schema validation failed for file /usr/bin/sg_modes (index: wazuh-states-fim-files). Errors: file.mtime: Invalid date format. Expected ISO8601, got: 2026-02-27T17:12:12.000Z
state.modified_at: Invalid date format. Expected ISO8601, got: 2026-07-14T04:47:54.799Z

Both timestamps are valid ISO8601, and the same event validates correctly at runtime — the error only appears during shutdown.

Root cause (shutdown race on process-static state): the FIM scan runs on the main thread (fim_scan) and validates each stateful event via validate_and_persist_fim_event()schema_validator_validate()isValidISO8601Date(), which uses a function-local static const std::regex. When a termination signal arrives, fim_shutdown_waiter() runs the teardown and calls HandleSIG(), which calls exit(1) (shared/src/sig_op.c). exit() destroys process-static objects — including that regex in libschema_validator — while the scan thread may still be validating. std::regex_match() on a destroyed regex is undefined behavior and, in practice, returns no-match, so every date check fails and a valid event is reported as invalid. Because the scan holds fim_scan_mutex, the waiter deliberately skips the FIM DB teardown but still proceeds to exit(), so the scan and static destruction run concurrently — this is why it only reproduces during shutdown.

Closes #37654

Proposed Changes

In validate_and_persist_fim_event() (src/syscheckd/src/run_check.c), when a schema-validation failure is observed while a shutdown is in progress (fim_shutdown_process_on()), treat the result as untrustworthy:

  • Log it at debug instead of error (no misleading ERROR during shutdown).
  • Do not act on the untrustworthy result: skip the DBSync deletion and the event persistence, and return true so the caller releases its resources without leaking. The agent is stopping and the event is re-evaluated on the next start.

Handling it at the point where the failure is detected (rather than skipping validation up front) also covers the case where the stop happens inside the validator, i.e. std::regex_match racing the static destruction mid-call.

Note on why deletion matters: in the scan path mark_for_deletion is set for INSERTED/MODIFIED files (src/syscheckd/src/file/file.c), so acting on a spurious shutdown-time failure could delete a valid entry from the FIM database (only to be re-reported on the next start). Returning early avoids that.

Results and Evidence

Before (from #37654, agent shutdown; the reported valid event is rejected):

2026/07/14 04:47:54 wazuh-syscheckd: INFO: (1756): Shutdown received. Releasing resources.
2026/07/14 04:47:54 wazuh-syscheckd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2026/07/14 04:47:54 wazuh-syscheckd: ERROR: Schema validation failed for file /usr/bin/sg_modes (index: wazuh-states-fim-files). Errors: file.mtime: Invalid date format. Expected ISO8601, got: 2026-02-27T17:12:12.000Z
state.modified_at: Invalid date format. Expected ISO8601, got: 2026-07-14T04:47:54.799Z
2026/07/14 04:47:54 wazuh-syscheckd: ERROR: Raw event that failed validation: {...}

After (with this change): no ERROR is emitted for a validation failure that happens during shutdown. When debug logging is enabled, a single debug line is logged instead, and the untrustworthy result is not acted on:

wazuh-syscheckd: DEBUG: Ignoring schema validation failure for file /usr/bin/sg_modes during shutdown: file.mtime: Invalid date format. Expected ISO8601, got: 2026-02-27T17:12:12.000Z ...

Genuine, non-shutdown validation failures are unchanged: they still log at ERROR with the failure reason and still skip persistence / mark for deletion.

Artifacts Affected

  • wazuh-syscheckd (agent). No changes to the schema validator library or its embedded schemas.

Configuration Changes

None.

Documentation Updates

None required. Recommendation: add the (now debug-level) shutdown message to the known-messages list of the epic (#37410) if needed.

Tests Introduced

No new unit tests: the behavior is a shutdown-timing race that is not exercised by the existing FIM unit suite, and no existing test asserts on the affected ERROR messages (verified). The change is a guarded log-level/early-return within validate_and_persist_fim_event(); the non-shutdown path is unchanged.

Review Checklist

  • Code changes reviewed
  • Relevant evidence provided
  • Tests cover the new functionality (n/a — shutdown-timing race; non-shutdown path unchanged)
  • Configuration changes documented (none)
  • Developer documentation reflects the changes
  • Meets requirements and/or definition of done
  • No unresolved dependencies with other issues

@Nicogp Nicogp linked an issue Jul 14, 2026 that may be closed by this pull request
@Nicogp
Nicogp merged commit 826eb0f into 5.0.0 Jul 14, 2026
69 of 70 checks passed
@Nicogp
Nicogp deleted the fix/37654-schema-validation-log-on-shutdown branch July 14, 2026 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Schema validation failed due to invalid date format

2 participants