Starting the elastic-agent docker container fails resolving ELASTICSEARCH_API_KEY variable even when it is provided

Allowing users to set the API key via an environment variable was introduced in #5536 which was first available in 9.0.0.

Any 9.0.0 and above container will fail to start with a failed: no matching vars: ${env.ELASTICSEARCH_API_KEY:} error even when the environment variable is provided.

docker run -e ELASTICSEARCH_API_KEY=test docker.elastic.co/elastic-agent/elastic-agent:9.0.0-SNAPSHOT
...
{"log.level":"error","@timestamp":"2025-08-12T20:17:36.203Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).runLoopIteration","file.name":"coordinator/coordinator.go","file.line":1213},"message":"applying new policy: generating component model: rendering outputs failed: rendering output \"default\" failed: no matching vars: ${env.ELASTICSEARCH_API_KEY:}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
Error: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/9.0/fleet-troubleshooting.html

The 8.19.0 container will fail to start with a similar error, but this time related to the ELASTICSEARCH_HOSTS variable as the ELASTICSEARCH_API_KEY variable does not exist in that branch:

docker run -e ELASTICSEARCH_HOSTS=testing docker.elastic.co/elastic-agent/elastic-agent:8.19.0-SNAPSHOT
...
{"log.level":"error","@timestamp":"2025-08-12T20:28:34.894Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).runLoopIteration","file.name":"coordinator/coordinator.go","file.line":1297},"message":"applying new policy: generating component model: rendering outputs failed: rendering output \"default\" failed: no matching vars: ${env.ELASTICSEARCH_HOSTS:http://elasticsearch:9200}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

@ycombinator

@ycombinator was able to determine that the change in this commit b59d51a#diff-efef2d2536ab052feff95a0b0565325806b28e733bb9f2abc319b646194d4de5 caused the error.

looking into determining why and the proper fix

The issue here is that the variables are not defined exactly the same as go-ucfg references them. go-ucfg allowed ${env.ELASTICSEARCH_API_KEY:} but the AST variable substitution requires ${env.ELASTICSEARCH_API_KEY:''}. The missing '' at the end is the issue.

This was an over sight on my part. I think allowing the AST to work with ${env.ELASTICSEARCH_API_KEY:} is the better solution as this would be non-breaking and doesn't require others to update there existing configuration. Which was the original goal of the b59d51a

The issue here is that the variables are not defined exactly the same as go-ucfg references them. go-ucfg allowed ${env.ELASTICSEARCH_API_KEY:} but the AST variable substitution requires ${env.ELASTICSEARCH_API_KEY:''}. The missing '' at the end is the issue.

Can we quickly add the missing '' to

elastic-agent/_meta/config/elastic-agent.docker.yml.tmpl

Lines 7 to 10 in e69dd4c

    
           hosts: '${ELASTICSEARCH_HOSTS:http://elasticsearch:9200}' 
        
           username: '${ELASTICSEARCH_USERNAME:}' 
        
           password: '${ELASTICSEARCH_PASSWORD:}' 
        
           api_key: '${ELASTICSEARCH_API_KEY:}'

? This would fix this immediately in a low effort way immediately with the default container configuration.

I think allowing the AST to work with ${env.ELASTICSEARCH_API_KEY:} is the better solution as this would be non-breaking and doesn't require others to update there existing configuration. Which was the original goal of the b59d51a

This makes sense as long as it is straight forward and doesn't have any other trade offs. 9.1.0 was released on July 29th so this hasn't been out in the wild for that long, we may see other reports of this later (though nobody internally has noticed it for 6 months).

@cmacknz

@cmacknz My comment was a little off. Once I dug more it was actually that it is using : and not |.

Navigation Menu

Repository navigation

Starting the elastic-agent docker container fails resolving ELASTICSEARCH_API_KEY variable even when it is provided #9328

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	hosts: '${ELASTICSEARCH_HOSTS:http://elasticsearch:9200}'
	username: '${ELASTICSEARCH_USERNAME:}'
	password: '${ELASTICSEARCH_PASSWORD:}'
	api_key: '${ELASTICSEARCH_API_KEY:}'

Starting the elastic-agent docker container fails resolving ELASTICSEARCH_API_KEY variable even when it is provided #9328

Description

Activity

elasticmachine commented on Aug 12, 2025

blakerouse commented on Aug 14, 2025

blakerouse commented on Aug 18, 2025

cmacknz commented on Aug 18, 2025

blakerouse commented on Aug 18, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions