Skip to content

Race in test TestResyncClearsStaleNamespaceIndex #28868

Open
Bug
Open
Race in test TestResyncClearsStaleNamespaceIndex#28868
Bug
Labels
bugSomething isn't workingbug/priority:highShould be fixed in the next patch releasecomponent:cacheIssue related to Kubernetes caching, corruption, or cache improvementscomponent:testTest requests (e2e or unit)triage/pendingThis issue needs further triage to be correctly classified

Description

@jannfis
jannfis
opened on Jul 22, 2026

Checklist:

  • I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
  • I've included steps to reproduce the bug.
  • I've pasted the output of argocd version.

Describe the bug

A race condition was detected in the unit tests of release-3.4. It seems to only fail intermittently, so not sure whether it affects 3.5 or latest master code too.

I0722 19:57:42.025994   14818 cluster.go:1010] "Start syncing cluster"
==================
WARNING: DATA RACE
Write at 0x00c0006d1790 by goroutine 498:
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).sync()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1022 +0x4af
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.TestResyncClearsStaleNamespaceIndex()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster_test.go:914 +0x10c5
  testing.tRunner()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2036 +0x21c
  testing.(*T).Run.gowrap1()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2101 +0x38

Previous read at 0x00c0006d1790 by goroutine 952:
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).onNodeRemoved()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1622 +0x171
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).processEvent()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1605 +0x1ec
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).recordEvent()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1546 +0x55e
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).watchEvents.func1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:859 +0xb6c
  github.com/argoproj/argo-cd/gitops-engine/pkg/utils/kube.RetryUntilSucceed.func1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/utils/kube/kube.go:459 +0xe5
  k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.34.0/pkg/util/wait/loop.go:53 +0xc5
  k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.34.0/pkg/util/wait/loop.go:54 +0x176
  k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.34.0/pkg/util/wait/poll.go:33 +0x6a
  github.com/argoproj/argo-cd/gitops-engine/pkg/utils/kube.RetryUntilSucceed()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/utils/kube/kube.go:457 +0x15a
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).watchEvents()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:794 +0x3b9
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).sync.func1.1.gowrap1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1114 +0x190

Goroutine 498 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2101 +0xb12
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2585 +0x85
  testing.tRunner()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2036 +0x21c
  testing.runTests()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2583 +0x9e9
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2443 +0xf4b
  main.main()
      _testmain.go:168 +0x164

Goroutine 952 (finished) created at:
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).sync.func1.1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1114 +0xabc
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).processApi()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:934 +0x342
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).sync.func1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1078 +0x3fb
  github.com/argoproj/argo-cd/gitops-engine/pkg/utils/kube.RunAllAsync.func1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/utils/kube/ctl.go:392 +0x38
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/sync@v0.17.0/errgroup/errgroup.go:93 +0x86
==================
==================
WARNING: DATA RACE
Write at 0x00c0006d1798 by goroutine 498:
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).sync()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1023 +0x514
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.TestResyncClearsStaleNamespaceIndex()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster_test.go:914 +0x10c5
  testing.tRunner()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2036 +0x21c
  testing.(*T).Run.gowrap1()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2101 +0x38

Previous read at 0x00c0006d1798 by goroutine 952:
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).onNodeRemoved()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1623 +0x1a4
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).processEvent()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1605 +0x1ec
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).recordEvent()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1546 +0x55e
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).watchEvents.func1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:859 +0xb6c
  github.com/argoproj/argo-cd/gitops-engine/pkg/utils/kube.RetryUntilSucceed.func1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/utils/kube/kube.go:459 +0xe5
  k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.34.0/pkg/util/wait/loop.go:53 +0xc5
  k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.34.0/pkg/util/wait/loop.go:54 +0x176
  k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.34.0/pkg/util/wait/poll.go:33 +0x6a
  github.com/argoproj/argo-cd/gitops-engine/pkg/utils/kube.RetryUntilSucceed()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/utils/kube/kube.go:457 +0x15a
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).watchEvents()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:794 +0x3b9
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).sync.func1.1.gowrap1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1114 +0x190

Goroutine 498 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2101 +0xb12
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2585 +0x85
  testing.tRunner()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2036 +0x21c
  testing.runTests()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2583 +0x9e9
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2443 +0xf4b
  main.main()
      _testmain.go:168 +0x164

Goroutine 952 (finished) created at:
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).sync.func1.1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1114 +0xabc
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).processApi()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:934 +0x342
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).sync.func1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1078 +0x3fb
  github.com/argoproj/argo-cd/gitops-engine/pkg/utils/kube.RunAllAsync.func1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/utils/kube/ctl.go:392 +0x38
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/sync@v0.17.0/errgroup/errgroup.go:93 +0x86
==================
==================
WARNING: DATA RACE
Write at 0x00c0006d1860 by goroutine 498:
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).sync()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1025 +0x5e4
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.TestResyncClearsStaleNamespaceIndex()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster_test.go:914 +0x10c5
  testing.tRunner()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2036 +0x21c
  testing.(*T).Run.gowrap1()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2101 +0x38

Previous read at 0x00c0006d1860 by goroutine 952:
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).removeFromParentUIDToChildren()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:567 +0x1024
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).onNodeRemoved()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1646 +0xe9f
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).processEvent()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1605 +0x1ec
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).recordEvent()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1546 +0x55e
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).watchEvents.func1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:859 +0xb6c
  github.com/argoproj/argo-cd/gitops-engine/pkg/utils/kube.RetryUntilSucceed.func1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/utils/kube/kube.go:459 +0xe5
  k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.34.0/pkg/util/wait/loop.go:53 +0xc5
  k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.34.0/pkg/util/wait/loop.go:54 +0x176
  k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.34.0/pkg/util/wait/poll.go:33 +0x6a
  github.com/argoproj/argo-cd/gitops-engine/pkg/utils/kube.RetryUntilSucceed()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/utils/kube/kube.go:457 +0x15a
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).watchEvents()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:794 +0x3b9
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).sync.func1.1.gowrap1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1114 +0x190

Goroutine 498 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2101 +0xb12
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2585 +0x85
  testing.tRunner()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2036 +0x21c
  testing.runTests()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2583 +0x9e9
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.26.0/x64/src/testing/testing.go:2443 +0xf4b
  main.main()
      _testmain.go:168 +0x164

Goroutine 952 (finished) created at:
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).sync.func1.1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1114 +0xabc
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).processApi()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:934 +0x342
  github.com/argoproj/argo-cd/gitops-engine/pkg/cache.(*clusterCache).sync.func1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/cache/cluster.go:1078 +0x3fb
  github.com/argoproj/argo-cd/gitops-engine/pkg/utils/kube.RunAllAsync.func1()
      /home/runner/work/argo-cd/argo-cd/gitops-engine/pkg/utils/kube/ctl.go:392 +0x38
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/sync@v0.17.0/errgroup/errgroup.go:93 +0x86
==================
I0722 19:57:42.030809   14818 cluster.go:1124] "Cluster successfully synced"
I0722 19:57:42.031817   14818 cluster.go:626] "Invalidated cluster"
--- FAIL: TestResyncClearsStaleNamespaceIndex (0.05s)
    testing.go:1712: race detected during execution of test
FAIL

To Reproduce

Expected behavior

Screenshots

Version

Paste the output from `argocd version` here.

Logs

Paste any relevant application logs here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingbug/priority:highShould be fixed in the next patch releasecomponent:cacheIssue related to Kubernetes caching, corruption, or cache improvementscomponent:testTest requests (e2e or unit)triage/pendingThis issue needs further triage to be correctly classified

    Type

    Bug

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions