Support EKS Pod Identity credentials in the S3 repository by somaz94 · Pull Request #151614 · elastic/elasticsearch

somaz94 · 2026-06-18T06:43:24Z

What

repository-s3 already supports EKS IRSA (web identity) but not EKS Pod Identity. With the ES 9.x entitlements model, the AWS SDK's ContainerCredentialsProvider reads the Pod Identity auth token from AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE (/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token), which is outside the repository-s3 entitlement-grantable area, so credential resolution fails with Failed to read token file.

How

This mirrors the IRSA web-identity setup: repository-s3 grants read access to a fixed entitled location, and the operator points the credential source at it.

entitlement-policy.yaml grants read on a fixed config-relative symlink repository-s3/eks-pod-identity-token, alongside the existing IRSA repository-s3/aws-web-identity-token-file grant.
The operator points AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE (or the aws.containerAuthorizationTokenFile system property) at that entitled path and symlinks the Kubernetes-injected token there, exactly as they already do for the IRSA web-identity token. S3Service does not override the env var or any system property, so there are no JVM-wide repercussions.

Token rotation needs no extra handling: ContainerCredentialsProvider re-reads the token on each resolution.

Testing

./gradlew :modules:repository-s3:compileJava :modules:repository-s3:compileJavaRestTestJava :modules:repository-s3:checkstyleMain :modules:repository-s3:forbiddenApisMain — BUILD SUCCESSFUL
./gradlew :modules:repository-s3:javaRestTest --tests "*RepositoryS3PodIdentityCredentialsRestIT" — BUILD SUCCESSFUL, 16 tests / 0 failures, run locally with Docker / testcontainers. The IT sets AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE to ${ES_PATH_CONF}/repository-s3/eks-pod-identity-token to mirror the operator setup.

Closes #106484

This change was implemented with the assistance of Claude Code (Opus 4.8). I have reviewed it and take responsibility for its correctness.

cla-checker-service · 2026-06-18T06:43:30Z

💚 CLA has been signed

elasticsearchmachine · 2026-06-18T18:29:16Z

Pinging @elastic/es-distributed (Team:Distributed)

DaveCTurner

Similar to the IRSA identity setup, we need to ask users to set this up themselves with env vars or system properties. We can't override system properties as you propose because this has JVM-wide repercussions.

somaz94 · 2026-06-19T02:26:56Z

Thanks, that makes sense. I've dropped the system-property override entirely.

The plugin no longer touches aws.containerAuthorizationTokenFile or AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE. It just keeps the read entitlement on the repository-s3/eks-pod-identity-token symlink, the same way the IRSA web-identity token works. Operators point AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE (or the aws.containerAuthorizationTokenFile sysprop) at that entitled path themselves and symlink the Kubernetes-injected token there.

So S3Service loses the maybeOverrideContainerAuthTokenFile / clearPodIdentitySysprop methods (about 100 lines removed), and the entitlement policy drops the write_system_properties grant for that sysprop. The IT now sets the env var to ${ES_PATH_CONF}/repository-s3/eks-pod-identity-token to mirror the operator setup, and :modules:repository-s3:javaRestTest passes locally (16 tests).

DaveCTurner

Thanks, ok, with all that gone it seems we could use repository-s3/aws-web-identity-token-file for EKS pod identity too, avoiding the need for this change entirely. Can you explain why the filename matters? Or am I missing something?

DaveCTurner

On reflection I think I accept the need for a different file name - it's not really the same thing, and it'd be confusing to overload this one location with two different meanings.

I left a couple of comments about the test, but overall it's doing pretty much the right thing.

The code comments aren't particularly useful (are they LLM-generated?) as they don't really describe the "why" of anything, only reiterating the "how". I think you could reasonably delete them, although comments linking to the relevant AWS docs might be useful in their places.

We also need to add to the reference manual the instructions about how to set up the relevant symlink, similar to the IRSA instructions here. Unfortunately that's in a different repository so will need a separate PR, but I'll be able to merge them together.

DaveCTurner · 2026-07-01T10:31:40Z

+    private static final String BASE_PATH = PREFIX + "base_path";
+    private static final String CLIENT = "pod_identity_credentials_client";
+
+    private static final String POD_IDENTITY_TOKEN_FILE_CONTENTS = "test-pod-identity-auth-token-" + UUID.randomUUID();


This UUID.randomUUID() is not a deterministic function of the test seed - I'd rather we stuck to using deterministic randomness. A little tricky here because deterministic randomness isn't available in static context so you'll need to use a Supplier<String> to delay the generation of this token until later when the test is running.

Done — the token is now produced lazily by a memoized Supplier<String> (LazyInitializable over randomIdentifier()) instead of UUID.randomUUID() in static context, so it's a deterministic function of the test seed and isn't evaluated until the test runs. The same memoized supplier feeds both the config file and the fixture's verification (see the other thread), so they always observe the same token.

DaveCTurner · 2026-07-01T10:43:29Z

+    private static final Supplier<String> regionSupplier = new DynamicRegionSupplier();
+    private static final DynamicAwsCredentials dynamicCredentials = new DynamicAwsCredentials(regionSupplier, "s3");
+
+    private static final Ec2ImdsHttpFixture podIdentityCredentialsFixture = new Ec2ImdsHttpFixture(


This appears to be approximately right but I'm not sure it's quite the same shape of response (e.g. does it include the RoleArn field?) and it doesn't verify that the token is correct.

Split into the two parts you raised:

Response shape: the credentials JSON the shared Ec2ImdsHttpHandler emits already includes RoleArn (alongside AccessKeyId/Expiration/SecretAccessKey/Token), which is what ContainerCredentialsProvider consumes — so the shape was already correct.

Token verification: added. Ec2ImdsServiceBuilder.authorizationTokenSupplier(...) makes the credentials endpoint require the exact Authorization header the SDK sends (the token it read from the entitled file) and return 403 otherwise, mirroring AwsStsHttpHandler. It's opt-in (null by default) so the other IMDS/ECS consumers are unaffected. The pod-identity test wires in the same memoized token, so it now genuinely verifies the SDK forwarded the entitled token. Also added a fixture unit test (testAlternativeCredentialsEndpointRequiresAuthorizationToken) covering the missing/wrong/correct-token cases, and confirmed the full RepositoryS3PodIdentityCredentialsRestIT still passes end-to-end.

the credentials JSON the shared Ec2ImdsHttpHandler emits already includes RoleArn

Indeed, but should it? Is this what the pod identity credentials endpoint returns? I couldn't find any docs confirming one way or the other.

You're right, it shouldn't be there. I checked the eks-pod-identity-agent source (EksCredentialsResponse in pkg/credentials/model.go) and the pod identity endpoint returns AccessKeyId, SecretAccessKey, Token, AccountId and Expiration, so an AccountId and no RoleArn. The RoleArn was just leaking in from reusing the shared IMDS/ECS handler.

I added a podIdentityCredentialsResponse() option to the fixture so this test now returns AccountId and drops RoleArn, with a unit test asserting the shape. Pushed as a new commit.

DaveCTurner · 2026-07-02T06:09:51Z

Thanks, I hope to get to this again next week. Please don't force-push to PRs under review -- we've lost the commits against which my earlier reviews were done.

somaz94 · 2026-07-02T06:54:58Z

Sorry about the force-push, I won't rebase this branch again while it's under review. Pushed the RoleArn fix above as a new commit on top instead.

Also opened the companion reference-manual PR you mentioned: elastic/docs-content#7182. It adds an EKS Pod Identity section next to the IRSA one.

somaz94 · 2026-07-02T08:59:55Z

Thanks for the review, @DaveCTurner. Pushed fc62a0e:

Trimmed the code comments — deleted the ones that just restated the adjacent code/asserts, and where a comment explains AWS-specific behavior I replaced the prose with a short note linking the EKS Pod Identity docs (pod-id-how-it-works.html).
The two inline threads were addressed earlier in 9e89b16: deterministic randomness now comes from randomIdentifier() (memoized via LazyInitializable, shared by the fixture and the config so both observe the same token) instead of UUID.randomUUID(); and the fixture returns the real Pod Identity response shape (AccountId, no RoleArn) with the credentials endpoint now verifying the Authorization token.

PTAL when you have a moment. The reference-manual docs are in the companion PR elastic/docs-content#7182.

DaveCTurner

LGTM

#7182) Documents how to use EKS Pod Identity credentials with the S3 snapshot repository, as a companion to elastic/elasticsearch#151614. Adds an "Using EKS Pod Identity for authentication" subsection alongside the existing IRSA instructions.

elasticsearchmachine added v9.5.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels Jun 18, 2026

somaz94 marked this pull request as ready for review June 18, 2026 07:09

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Jun 18, 2026

john-wagster added the :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs label Jun 18, 2026

elasticsearchmachine added Team:Distributed Meta label for distributed team. and removed needs:triage Requires assignment of a team area label labels Jun 18, 2026

This comment was marked as off-topic.

Sign in to view

DaveCTurner requested changes Jun 18, 2026

View reviewed changes

DaveCTurner requested changes Jul 1, 2026

View reviewed changes

DaveCTurner reviewed Jul 1, 2026

View reviewed changes

DaveCTurner added the >enhancement label Jul 1, 2026

DaveCTurner self-assigned this Jul 1, 2026

somaz94 added 4 commits July 2, 2026 12:02

Support EKS Pod Identity credentials in the S3 repository

f77ea48

Add changelog entry for EKS Pod Identity S3 repository support

28cc9dc

Drop the system-property override for EKS Pod Identity

ed5735a

Verify the SDK forwards the entitled EKS Pod Identity token

81d02b5

somaz94 force-pushed the repository-s3-eks-pod-identity branch from d05a344 to 81d02b5 Compare July 2, 2026 03:10

somaz94 mentioned this pull request Jul 2, 2026

Add EKS Pod Identity authentication instructions for the S3 repository elastic/docs-content#7182

Merged

1 task

Make the pod identity test fixture return AccountId instead of RoleArn

9e89b16

Trim review comments and link EKS Pod Identity docs

fc62a0e

DaveCTurner added 2 commits July 7, 2026 10:43

Misc cleanups

1a066b8

Merge branch 'main' into repository-s3-eks-pod-identity

c88c0d9

DaveCTurner approved these changes Jul 7, 2026

View reviewed changes

elasticsearchmachine added v9.6.0 and removed v9.5.0 labels Jul 8, 2026

DaveCTurner merged commit b5cdd68 into elastic:main Jul 9, 2026
38 checks passed

DaveCTurner mentioned this pull request Jul 10, 2026

Add EKS Pod Identity authn docs for S3 repository elastic/docs-content#7316

Closed

2 tasks

Conversation

somaz94 commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Testing

Uh oh!

cla-checker-service Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 18, 2026

Uh oh!

This comment was marked as off-topic.

DaveCTurner left a comment

Choose a reason for hiding this comment

Uh oh!

somaz94 commented Jun 19, 2026

Uh oh!

DaveCTurner left a comment

Choose a reason for hiding this comment

Uh oh!

DaveCTurner left a comment

Choose a reason for hiding this comment

Uh oh!

DaveCTurner Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

somaz94 Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

DaveCTurner Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

somaz94 Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

DaveCTurner Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

somaz94 Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

DaveCTurner commented Jul 2, 2026

Uh oh!

somaz94 commented Jul 2, 2026

Uh oh!

somaz94 commented Jul 2, 2026

Uh oh!

DaveCTurner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

somaz94 commented Jun 18, 2026 •

edited

Loading

cla-checker-service Bot commented Jun 18, 2026 •

edited

Loading