Checklist

• I searched existing Argo CD issues for server-side diff false negatives involving removeWebhookMutation and x-kubernetes-preserve-unknown-fields.
• I reproduced the behavior with Argo CD v3.4.4 comparison code and a Kubernetes server-side dry-run result.
• The desired and live resources differ in a field owned by argocd-controller.
• The problem disappears with IncludeMutationWebhook=true or ServerSideDiff=false.

Version

Argo CD v3.4.4
Server-side diff enabled
Server-side apply enabled

Summary

Server-side diff can return Modified=false for a custom resource whose desired spec differs from live, even though:

• Kubernetes server-side apply dry-run correctly predicts the desired spec;
• metadata.managedFields says argocd-controller owns the changed field; and
• no ignoreDifferences rule applies.

The false negative occurs in removeWebhookMutation. An OpenAPI-derived parent field path is classified as non-Argo because the managed-fields set contains owned descendants but not the parent path itself. Removing the parent also removes those owned descendants. Merging the result into live restores the old spec, so the final comparison incorrectly reports equality.

Minimal schema shape

The problem is triggered by a structural CRD containing a preserved object with known children. For example:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: examples.example.io
spec:
  group: example.io
  scope: Namespaced
  names:
    plural: examples
    singular: example
    kind: Example
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
              properties:
                configuration:
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
                  properties:
                    value:
                      type: string

Representative live state:

apiVersion: example.io/v1
kind: Example
metadata:
  name: sample
  namespace: default
spec:
  configuration:
    value: old

Representative desired state:

apiVersion: example.io/v1
kind: Example
metadata:
  name: sample
  namespace: default
spec:
  configuration:
    value: new

The resource is initially applied using server-side apply and field manager argocd-controller. A subsequent server-side dry-run with the desired state correctly returns value: new.

Relevant field sets

For the predicted object, the GVK parser's ToFieldSet() contains parent and descendant paths similar to:

.spec
.spec.configuration
.spec.configuration.value

The Kubernetes-generated managed-fields entry for argocd-controller contains the owned descendant, but not every parent container path:

.spec.configuration.value

removeWebhookMutation currently computes:

nonArgoFieldsSet := predictedLiveFieldSet.Difference(managerFieldsSet)
typedPredictedLive = typedPredictedLive.RemoveItems(nonArgoFieldsSet)
typedPredictedLive, err = typedLive.Merge(typedPredictedLive)

The difference retains .spec and .spec.configuration. RemoveItems therefore removes the complete desired subtree, including .spec.configuration.value, even though that descendant is owned by argocd-controller. Merging into live restores value: old.

Reproduction results

Using the same config, live object, predicted server-side dry-run JSON, OpenAPI schema, and manager name with gitops-engine/pkg/diff.ServerSideDiff:

default behavior (webhook mutations filtered):
  Modified=false
  predicted spec after filtering equals the stale live spec

WithIgnoreMutationWebhook(false), equivalent to IncludeMutationWebhook=true:
  Modified=true
  predicted spec after filtering retains the desired value

The server-side dry-run output itself is correct. The false negative is introduced after the dry-run by mutation-removal processing.

This code path runs by default for server-side diff; it does not require Argo CD to receive an explicit signal that a mutating webhook changed the object. A mutating webhook that adds metadata makes this path operationally relevant, but it does not need to modify the spec.

Expected behavior

Server-side diff should return Modified=true whenever an argocd-controller-owned descendant differs between the predicted and live objects, regardless of whether an ancestor path is absent from the manager's field set.

Filtering webhook mutations must not remove manager-owned descendants.

Actual behavior

Server-side diff returns Modified=false. With selective sync (ApplyOutOfSyncOnly=true), the resource is omitted from the sync task list and can remain stale indefinitely.

Disabling selective sync forces an apply and makes the live resource converge, but the next comparison can still produce the same false negative.

Workarounds

Any of these avoids the convergence failure:

metadata:
  annotations:
    argocd.argoproj.io/compare-options: IncludeMutationWebhook=true

or:

metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=false

Removing ApplyOutOfSyncOnly=true also forces convergence, but does not correct the comparison result.

Possible fix direction

Mutation filtering should be ancestor-aware. A parent path should not be removed when it contains any descendant owned by the selected manager.

Another option is to start from the paths that actually differ between predicted and live and filter only changed paths that are not covered by the manager's field set, instead of removing every predicted field absent from the manager's field set.

A regression test should include:

• a CRD with x-kubernetes-preserve-unknown-fields: true on an object containing known child properties;
• a predicted field set containing both the parent and descendants;
• a managed-fields entry containing an owned descendant without the parent path;
• a desired/live difference at that owned descendant; and
• assertions for both default mutation filtering and IncludeMutationWebhook=true.

Impact

This is a silent convergence failure rather than a cosmetic diff problem. Applications can report Synced while live custom resources retain old desired state. When combined with ApplyOutOfSyncOnly=true, automated sync and self-heal repeatedly skip the affected resources.