[beats receivers] Connection level output 401 error handling does not match the process runtime

Relates fix(otelconsumer): do not retry 401 errors from Elasticsearch beats#50261
Relates Standalone Agent data loss due to 401 errors not being retried cloud-on-k8s#9406
Depends [exporter/elasticsearchexporter] Allow configuring separate request and document level retry policies open-telemetry/opentelemetry-collector-contrib#48681

In the beats Elasticsearch output, connection level errors like 401s are always retried:

https://github.com/elastic/beats/blob/71d0f09b540201d44155615d01a35639895d64f3/libbeat/outputs/elasticsearch/client.go#L255-L264

	// Create and send the bulk request.
	bulkResult := client.doBulkRequest(ctx, batch)
	span.Context.SetLabel("events_encoded", len(bulkResult.events))
	if bulkResult.connErr != nil {
		// If there was a connection-level error there is no per-item response,
		// handle it and return.
		return client.handleBulkResultError(ctx, batch, bulkResult)
	}
	span.Context.SetLabel("events_published", len(bulkResult.events))

However, most document level 4xx codes like 401s are not retried:

https://github.com/elastic/beats/blob/71d0f09b540201d44155615d01a35639895d64f3/libbeat/outputs/elasticsearch/client.go#L491-L523

	encodedEvent := event.EncodedEvent.(*encodedEvent) //nolint:errcheck //safe to ignore type check
	if itemStatus < 300 {
		if encodedEvent.deadLetter {
			// This was ingested into the dead letter index, not the original target
			stats.deadLetter++
		} else {
			stats.acked++
		}
		return false // no retry needed
	}

	if itemStatus == 409 {
		// 409 is used to indicate there is already an event with the same ID, or
		// with identical Time Series Data Stream dimensions when TSDS is active.
		stats.duplicates++
		return false // no retry needed
	}

	if itemStatus == http.StatusTooManyRequests {
		stats.fails++
		stats.tooMany++
		return true
	}

	if itemStatus < 500 {

In Elastic Agent, the ES output translation configures the following set of retryable status codes which excludes 401s:

elastic-agent/internal/pkg/otel/translate/output_elasticsearch.go

Lines 38 to 53 in 902c65b

    
           RetryOnStatus: []int{ 
        
           	// 429 
        
           	http.StatusTooManyRequests, 
        
           	// 5xx 
        
           	http.StatusInternalServerError, 
        
           	http.StatusNotImplemented, 
        
           	http.StatusBadGateway, 
        
           	http.StatusServiceUnavailable, 
        
           	http.StatusGatewayTimeout, 
        
           	http.StatusHTTPVersionNotSupported, 
        
           	http.StatusVariantAlsoNegotiates, 
        
           	http.StatusInsufficientStorage, 
        
           	http.StatusLoopDetected, 
        
           	http.StatusNotExtended, 
        
           	http.StatusNetworkAuthenticationRequired, 
        
           },

This difference is causing problems in other systems like ECK where TLS certificate propagation has some delay, leading to data loss on the first events shipped after startup. See elastic/cloud-on-k8s#9406

In the Elasticsearch exporter, the same RetryOnStatus configuration applies to both request and document level retries making it so that we can't exactly match the beats behavior which would retry indefinitely for completely invalid credentials but not stall the pipeline because of a single missing index privilege in an otherwise valid API key (which would be a document level 401):

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/58b9c584e9cf737e68d7b5a0521ed43c5d57db65/exporter/elasticsearchexporter/config.go#L274-L275

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

Short term it is probably better to retry on request and document level 401+403 if we can fix this quickly.

It does look like the ES exporter would allow setting separate request and document level retry configurations:

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/58b9c584e9cf737e68d7b5a0521ed43c5d57db65/exporter/elasticsearchexporter/bulkindexer.go#L98-L102

	return docappender.BulkIndexerConfig{
		Client:                  client,
		MaxDocumentRetries:      maxDocRetries,
		Pipeline:                config.Pipeline,
		RetryOnDocumentStatus:   config.Retry.RetryOnStatus,

and https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/58b9c584e9cf737e68d7b5a0521ed43c5d57db65/exporter/elasticsearchexporter/esclient.go#L243-L247

		RetryOnStatus: config.Retry.RetryOnStatus,
		DisableRetry:  !config.Retry.Enabled,
		RetryOnError: func(_ *http.Request, err error) bool {
			return !errors.Is(err, context.Canceled) && !errors.Is(err, context.DeadlineExceeded)
		},

tl;dr I can reproduce the mismatch in code: the OTel Elasticsearch output translation used by beats-receiver/OTel runtime explicitly excludes 401 from retryable status codes, while the process runtime path still uses libbeat output handling that retries connection-level failures.

Recommendation
Align beats-receiver/OTel runtime behavior with process runtime for connection-level 401 by updating the OTel ES translation retry policy and adding a regression test. Concretely, update internal/pkg/otel/translate/output_elasticsearch.go so retry behavior for 401 matches the agreed policy for process runtime parity, and lock it with tests in output_elasticsearch_test.go + otelconfig_test.go.

Findings

OTel translation currently hard-codes retry statuses to 429 + 5xx only:
- internal/pkg/otel/translate/output_elasticsearch.go:38-53 (RetryOnStatus default list; no 401)
- internal/pkg/otel/translate/output_elasticsearch.go:180-188 (writes retry.retry_on_status from that list)
Tests assert and preserve this behavior today:
- internal/pkg/otel/translate/output_elasticsearch_test.go:59-71
- internal/pkg/otel/translate/otelconfig_test.go:291-297
Runtime paths are split, so this translation only affects OTel-managed components:
- internal/pkg/agent/application/coordinator/coordinator.go:2035-2042 (runtime model + otel model split/update)
- internal/pkg/agent/application/coordinator/coordinator.go:2199-2207 (splitModelBetweenManagers by runtime manager)
Process runtime output unit still passes output config directly (no OTel translation layer):
- pkg/component/component.go:605-612 (ExpectedConfig(output.Config) for output unit)
In libbeat Elasticsearch client, connection-level errors are retried via handleBulkResultError, while document-level <500 statuses are generally non-retry (except 429):
- beats/libbeat/outputs/elasticsearch/client.go:258-261, 327-359
- beats/libbeat/outputs/elasticsearch/client.go:519-526
Related upstream context:
- elastic/beats#50261 (merged): “fix(otelconsumer): do not retry 401 errors from Elasticsearch”

Verification

I validated current behavior by running targeted translation tests.

$ cd /home/runner/work/elastic-agent/elastic-agent
$ go test ./internal/pkg/otel/translate -run 'TestToOtelConfig|TestCompressionConfig|TestToOTelConfig_CheckUnsupported'
go: go.mod requires go >= 1.26.3 (running go 1.25.10; GOTOOLCHAIN=local)

$ cd /home/runner/work/elastic-agent/elastic-agent
$ GOTOOLCHAIN=auto go test ./internal/pkg/otel/translate -run 'TestToOtelConfig|TestCompressionConfig|TestToOTelConfig_CheckUnsupported'
ok   github.com/elastic/elastic-agent/internal/pkg/otel/translate 0.230s

$ cd /home/runner/work/elastic-agent/elastic-agent
$ GOTOOLCHAIN=auto go test ./internal/pkg/otel/translate -run TestUnitToExporterConfig
ok   github.com/elastic/elastic-agent/internal/pkg/otel/translate 0.312s

These passing tests currently encode the default retry list without 401.

Detailed Action Plan

Update OTel ES translation retry defaults in internal/pkg/otel/translate/output_elasticsearch.go:
- Modify defaultOptions.RetryOnStatus (around L38-L53) to match the intended parity policy for connection-level 401.
Update translation expectations in tests:
- internal/pkg/otel/translate/output_elasticsearch_test.go expected YAML retry_on_status lists.
- internal/pkg/otel/translate/otelconfig_test.go:291-297 expected retry_on_status list.
Add a focused regression test that asserts parity intent explicitly:
- In output_elasticsearch_test.go, add a test case that validates presence/absence of 401 in translated retry policy per the decided behavior.
Validate with targeted package tests:
- GOTOOLCHAIN=auto go test ./internal/pkg/otel/translate -run 'TestToOtelConfig|TestUnitToExporterConfig'
Optional hardening (if team wants stronger guardrail):
- Add a coordinator/runtime-level test that ensures OTel-managed output retry policy remains aligned with process-runtime policy for 401 handling assumptions.

Related Items

Type	Link / File	Relevance
Issue	#14531	Current triage target
PR	elastic/beats#50261	Upstream change touching 401 retry behavior in otelconsumer
File	`internal/pkg/otel/translate/output_elasticsearch.go:38-53`	Default retryable statuses (no 401)
File	`internal/pkg/otel/translate/output_elasticsearch.go:180-188`	Maps retry config into OTel exporter
File	`internal/pkg/otel/translate/output_elasticsearch_test.go:59-71`	Test fixtures asserting current retry list
File	`internal/pkg/otel/translate/otelconfig_test.go:291-297`	Expected retry list in unit-to-exporter mapping
File	`pkg/component/component.go:605-612`	Process runtime output unit config path
File	`internal/pkg/agent/application/coordinator/coordinator.go:2035-2042`	Runtime/OTel split update path
File	`beats/libbeat/outputs/elasticsearch/client.go:258-261`	Connection-level error handling path
File	`beats/libbeat/outputs/elasticsearch/client.go:519-526`	Per-item non-retry behavior for `<500` except 429

Note

🔒 Integrity filter blocked 3 items

The following items were blocked because they don't meet the GitHub integrity level.

#401 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#50217 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#9406 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Linking to the upstream feature request open-telemetry/opentelemetry-collector-contrib#48681

When we implement this on the agent side, let's make sure we have a test that the exponential backoff on 401 works with the expected delay increase. There is an active support case right now where versions before 9.3.5 retry immediately causing force unenrolled agents to hammer Elasticsearch and we want to avoid introducing that bug again.

@cmacknz

@cmacknz should we maintain the current status code retried in Otel mode and add 401 and 403 for request level retries, or should we fully match the Elasticsearch output and retry anything >= 500 at document level? Currently I'm going with the first approach, but that's an easy change.

We should exactly match the beats implementation, so if I've specified anything that conflicts with that go with the beats implementation (unless there is a flaw in it, then we should fix both variants).

fix(otel): separate Elasticsearch request and document retries (#16089)

fix(otel): separate Elasticsearch request and document retries (#16089)…

[8.19] (backport #16089) fix(otel): separate Elasticsearch request an…

[9.4] (backport #16089) fix(otel): separate Elasticsearch request and…

Navigation Menu

Repository navigation

[beats receivers] Connection level output 401 error handling does not match the process runtime #14531

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	RetryOnStatus: []int{
	// 429
	http.StatusTooManyRequests,
	// 5xx
	http.StatusInternalServerError,
	http.StatusNotImplemented,
	http.StatusBadGateway,
	http.StatusServiceUnavailable,
	http.StatusGatewayTimeout,
	http.StatusHTTPVersionNotSupported,
	http.StatusVariantAlsoNegotiates,
	http.StatusInsufficientStorage,
	http.StatusLoopDetected,
	http.StatusNotExtended,
	http.StatusNetworkAuthenticationRequired,
	},

[beats receivers] Connection level output 401 error handling does not match the process runtime #14531

Description

Activity

cmacknz commented on May 25, 2026

infra-vault-gh-plugin-prod commented on May 25, 2026

infra-vault-gh-plugin-prod commented on May 25, 2026

cmacknz commented on May 25, 2026

github-actions commented on May 25, 2026

cmacknz commented on May 26, 2026

cmacknz commented on Jul 14, 2026

belimawr commented on Aug 7, 2026

cmacknz commented on Aug 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions