Skip to content

fix(config): use actual file modification time in UpdateLastFileModTime#1668

Open
RaminGe wants to merge 6 commits into
TwiN:masterfrom
RaminGe:fix/issue-1657-config-watcher-clock-skew
Open

fix(config): use actual file modification time in UpdateLastFileModTime#1668
RaminGe wants to merge 6 commits into
TwiN:masterfrom
RaminGe:fix/issue-1657-config-watcher-clock-skew

Conversation

@RaminGe

@RaminGe RaminGe commented May 14, 2026

Copy link
Copy Markdown

Summary

Fixes #1657

Previously, UpdateLastFileModTime() set the last file modification time to time.Now(). In containerized environments with volume mounts (e.g. Kubernetes, Google Cloud Run), the mounted file's timestamp can be in the future relative to the container's internal clock. This caused the configuration watcher to continuously evaluate that the file had been modified (because config.lastFileModTime < fileInfo.ModTime()), triggering infinite restart loops and effectively disabling the alerting engine.

This PR updates UpdateLastFileModTime() to read the actual ModTime of the config file using os.Stat (and if it is a directory, using the newest ModTime inside that directory). It falls back to time.Now() only if the true modification time cannot be determined.

Checklist

  • Tested and/or added tests to validate that the changes work as intended, if applicable. -> Tested locally
  • Updated documentation in README.md, if applicable. -> Not applicable

Fixes TwiN#1657

Previously, UpdateLastFileModTime() set the last file modification time to time.Now(). In containerized environments with volume mounts (e.g. Kubernetes, Google Cloud Run), the mounted file's timestamp can be in the future relative to the container's internal clock. This caused the configuration watcher to continuously evaluate that the file had been modified (because config.lastFileModTime < fileInfo.ModTime()), triggering infinite restart loops and effectively disabling the alerting engine.

This commit updates UpdateLastFileModTime() to read the actual ModTime of the config file using os.Stat (and if it is a directory, using the newest ModTime inside that directory). It falls back to time.Now() only if the true modification time cannot be determined.
@github-actions github-actions Bot added the bug Something isn't working label May 14, 2026
@mgenoni

mgenoni commented May 14, 2026

Copy link
Copy Markdown

@RaminGe 🐎🔥

@TwiN TwiN left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Thank you for the contribution!

Comment thread config/config.go Outdated
Comment thread config/config.go Outdated
@TwiN

TwiN commented May 19, 2026

Copy link
Copy Markdown
Owner

@RaminGe is there any chance you could add a test for your change?

@RaminGe

RaminGe commented May 19, 2026

Copy link
Copy Markdown
Author

@RaminGe is there any chance you could add a test for your change?

@TwiN done!

@RaminGe RaminGe force-pushed the fix/issue-1657-config-watcher-clock-skew branch from 9694312 to e02d076 Compare May 20, 2026 09:05
@RaminGe

RaminGe commented Jun 1, 2026

Copy link
Copy Markdown
Author

Hey @TwiN anything else needed on this? Would really appreciate a merge as we use Gatus for our production system monitoring and our custom alerting workaround sucks

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes an infinite configuration hot-reload loop in containerized deployments by making Config.UpdateLastFileModTime() record the actual filesystem modification time (including directory-based configs), rather than using time.Now(). This aligns the watcher’s baseline with the mounted file’s timestamp (even if it is “in the future” relative to the container clock), preventing repeated false-positive “modified” detections that disrupt monitoring/alerting.

Changes:

  • Update UpdateLastFileModTime() to use os.Stat(...).ModTime() (or the newest YAML file ModTime within a config directory).
  • Add unit tests covering single-file, directory, and missing-file fallback behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
config/config.go Computes lastFileModTime from actual file/dir ModTime instead of time.Now(), preventing false reload loops.
config/config_test.go Adds test coverage for the new ModTime behavior across file/dir/missing-path cases.

Comment thread config/config_test.go
Comment on lines +2635 to +2641
dir := t.TempDir()
configFilePath := filepath.Join(dir, "config.yaml")
_ = os.WriteFile(configFilePath, []byte(`endpoints:
- name: test
url: https://example.com
`), 0o644)
time.Sleep(10 * time.Millisecond)
Comment thread config/config_test.go
Comment on lines +2646 to +2650
config.UpdateLastFileModTime()
stat, _ := os.Stat(configFilePath)
if config.lastFileModTime != stat.ModTime() {
t.Errorf("expected lastFileModTime to be %v, got %v", stat.ModTime(), config.lastFileModTime)
}
Comment thread config/config_test.go
Comment on lines +2656 to +2660
configDir.UpdateLastFileModTime()
stat, _ := os.Stat(configFilePath)
if configDir.lastFileModTime != stat.ModTime() {
t.Errorf("expected lastFileModTime to be %v, got %v", stat.ModTime(), configDir.lastFileModTime)
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Constant restart loop triggered by listenToConfigurationFileChanges preventing Slack alerts

4 participants