Skip to content

[Windows][FIPS] Upgrade from 9.5.0-SNAPSHOT to 9.6.0-SNAPSHOT fails because .elastic-agent.active.commit is missing. #16050

Description

@harshitgupta-qasource

Kibana Build details:

VERSION: 9.6.0-SNAPSHOT
BUILD: 108060
COMMIT: f86929ed091de600b6a2a9c20cf6e8e63604318c

Host OS and Browser version: Windows x64, All browsers

Preconditions:

  1. A 9.6.0-SNAPSHOT Kibana cloud environment is available.
  2. A Windows x64 endpoint is available.
  3. Elastic Agent 9.5.0-SNAPSHOT 64-bit (x86_64) FIPS artifact has been downloaded.
  4. The FIPS Elastic Agent has been installed on the Windows x64 endpoint and enrolled in Fleet.
  5. The agent is Healthy and FIPS mode is enabled.
  6. Elastic Agent 9.6.0-SNAPSHOT is available as an upgrade version.

Steps to reproduce:

  1. Navigate to Fleet → Agents.
  2. Select the enrolled Windows FIPS agent.
  3. Click Actions → Upgrade agent.
  4. Select 9.6.0-SNAPSHOT as the target version.
  5. Start the upgrade.
  6. Wait for the upgrade operation to complete.
  7. Observe that the upgrade transitions to the Failed state.
  8. Navigate to the agent's Logs tab.
  9. Observe that Windows FIPS Elastic Agent fails to upgrade from 9.5.0-SNAPSHOT to 9.6.0-SNAPSHOT and stuck in the Failed state.
  • The following error is observed in the agent logs:
upgrade to version 9.6.0-SNAPSHOT failed: reading metadata for elastic agent version 9.6.0-SNAPSHOT package "C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-9.5.0-SNAPSHOT-7feed3\\downloads\\elastic-agent-fips-9.6.0-SNAPSHOT-windows-x86_64.zip": looking up ".elastic-agent.active.commit" in package: open elastic-agent-fips-9.6.0-SNAPSHOT-windows-x86_64/.elastic-agent.active.commit: file does not exist

Expected Result:

  • The Windows FIPS Elastic Agent should upgrade successfully from 9.5.0-SNAPSHOT to 9.6.0-SNAPSHOT.

Whats Working Fine

For NON-FIPS windows agent upgrade successfully

Image

Screenshots:

  1. The missing .elastic-agent.active.commit error in the agent logs.
Image
  1. The agent remaining on version 9.5.0 with the "Upgrade failed" status and FIPS mode enabled.
Image

Agent logs:

elastic-agent-diagnostics-2026-08-06T07-27-04Z-00.zip

Ticket:

https://github.com/elastic/ingest-dev/issues/7431

Activity

  1. infra-vault-gh-plugin-prod commented on Aug 6, 2026

    @infra-vault-gh-plugin-prod

    Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

  2. harshitgupta-qasource commented on Aug 6, 2026

    @harshitgupta-qasource
    Author

    @kishorkumar-qasource Kindly review

  3. kishorkumar-qasource commented on Aug 6, 2026

    @kishorkumar-qasource

    Secondary review is done for this!!

  4. ebeahan commented on Aug 6, 2026

    @ebeahan
    Member

    Claude-Generated Root Cause Analysis

    Primary Code Bug — -fips not stripped from zip fileNamePrefix

    The codebase has two separate paths for reading package metadata: one for .tar.gz (Linux/macOS) and one for .zip (Windows). They handle the -fips filename substring inconsistently.

    For .tar.gz — getFileNamePrefix in step_unpack.go:662:

    func getFileNamePrefix(archivePath string) string {
        prefix := strings.TrimSuffix(filepath.Base(archivePath), ".tar.gz") + "/"
        prefix = strings.Replace(prefix, fipsPrefix, "", 1)  // ← strips "-fips"
        return prefix
    }

    For elastic-agent-fips-9.6.0-SNAPSHOT-linux-x86_64.tar.gz → prefix becomes elastic-agent-9.6.0-SNAPSHOT-linux-x86_64/.

    For .zip — getPackageMetadataFromZip (step_unpack.go:269) and unzip (step_unpack.go:128):

    fileNamePrefix := strings.TrimSuffix(filepath.Base(archivePath), ".zip") + "/"
    // ← NO "-fips" stripping

    For elastic-agent-fips-9.6.0-SNAPSHOT-windows-x86_64.zip → prefix becomes elastic-agent-fips-9.6.0-SNAPSHOT-windows-x86_64/.

    The Windows FIPS zip archive follows the same convention as tar.gz packages: the internal directory omits -fips (i.e., elastic-agent-9.6.0-SNAPSHOT-windows-x86_64/). The code then tries to open elastic-agent-fips-9.6.0-SNAPSHOT-windows-x86_64/.elastic-agent.active.commit, which does not exist because the actual path inside the archive is elastic-agent-9.6.0-SNAPSHOT-windows-x86_64/.elastic-agent.active.commit. This is confirmed precisely by the error message:

    open elastic-agent-fips-9.6.0-SNAPSHOT-windows-x86_64/.elastic-agent.active.commit: file does not exist
    

    The manifest lookup (step_unpack.go:336–349) silently ignores fs.ErrNotExist, so the wrong prefix for the manifest does not surface as an error. The commit file lookup (step_unpack.go:352–356) treats any error including fs.ErrNotExist as fatal — so the wrong prefix causes a hard failure here.

    Why non-FIPS Windows works: the archive name elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip already matches the internal directory name, so no stripping is needed and no prefix mismatch occurs.


    Secondary Potential Cause — Build/packaging omission

    It is also possible the FIPS Windows .zip artifact was built without .elastic-agent.active.commit at any path (i.e., the file is absent from the archive entirely rather than present at the wrong path). This would produce the same error. It is not possible to confirm without inspecting the actual artifact, but the code asymmetry above is the more likely cause.


    Contributing Factor — No FIPS zip test coverage

    TestGetFileNamePrefix (step_unpack_test.go:760) only tests .tar.gz paths. All unzip tests use non-FIPS archive names (e.g., elastic-agent-1.2.3-SNAPSHOT-someos-x86_64.zip). There is no test for getPackageMetadataFromZip or unzip with a FIPS-named .zip file, so the prefix mismatch was never caught.


    Fix Direction

    Apply the same -fips stripping to the zip code path. Both getPackageMetadataFromZip and unzip should strip -fips when computing the prefix, for example:

    fileNamePrefix := strings.TrimSuffix(strings.Replace(filepath.Base(archivePath), fipsPrefix, "", 1), ".zip") + "/"

    Or factor the logic into a shared helper analogous to getFileNamePrefix that handles both .zip and .tar.gz extensions.

  5. self-assigned this
    on Aug 7, 2026
  6. harshitgupta-qasource commented on Sep 8, 2026

    @harshitgupta-qasource
    Author

    Hi Team,

    We have validated this ticket on the latest 9.5.3 Kibana cloud environment and found it working fine.

    Observations:

    • Windows FIPS Elastic Agent should upgrade successfully from 9.5.2 to 9.5.3

    Build details:

    VERSION: 9.5.3 
    BUILD: 107530
    COMMIT: 2169f1de4c917fce03905cc3110f3f523e2184d2
    Artifact: https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-fips-9.5.2-windows-arm64.zip
    

    Screenshot

    • Before Upgrade
    Image
    • After Upgrade
    20260908-1014-11.3057709.mp4

    Hence, we are marking this issue as QA: Validated.
    Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

QA:ValidatedValidated by the QA TeamTeam:Elastic-Agent-Control-PlaneLabel for the Agent Control Plane teambugSomething isn't workingimpact:highShort-term priority; add to current release, or definitely next.

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions