Hi dear community,

I've installed fluentbit helm chart on K8s (AWS EKS) and I am working with IAM roles for service accounts (this way) to send logs to aws opensearch service.
When telling to the fluentbit deployment to work with the serviceaccount that map my role on aws It seems it is looking for a /var/run/secrets/eks.amazonaws.com/serviceaccount/aws-iam-token file to get the token:

but the default and mounted path is /var/run/secrets/eks.amazonaws.com/serviceaccount/token:

Then it cannot fetch the credentials to assume the role ..
In somehow when creating the role and the service account, the env variable injected is AWS_WEB_IDENTITY_TOKEN_FILE : /var/run/secrets/eks.amazonaws.com/serviceaccount/token but the pod look for /var/run/secrets/eks.amazonaws.com/serviceaccount/aws-iam-token

as a result I got this error on the fluentbit pod logs:

[2024/03/27 10:46:29] [error] [aws_credentials] STS assume role request failed
[2024/03/27 10:46:29] [ warn] [aws_credentials] No cached credentials are available and a credential refresh is already in progress. The current co-routine will retry.
[2024/03/27 10:46:29] [error] [signv4] Provider returned no credentials, service=es
[2024/03/27 10:46:29] [error] [output:opensearch:opensearch.0] could not sign request with sigv4
[2024/03/27 10:46:29] [ warn] [engine] chunk '1-1711536378.324856511.flb' cannot be retried: task_id=22, input=tail.0 > output=opensearch.0
[2024/03/27 10:46:29] [ info] [input] tail.0 resume (mem buf overlimit)

I understand ths is a known issue but when checking, it is not clear how this can be solved:

• Here it says:

Yeah, basically it's because the config map sets "" as the default for aws_sts_endpoint instead of NULL. This leads the code to incorrectly think that there is an custom STS endpoint, and then Fluent Bit tries to make a request to "".
https://github.com/fluent/fluent-bit/blob/master/plugins/out_es/es.c#L804

But that issue about fluentbit from app code perspective getting "" is supposed to be fixed now (I am using v2.2.2 )
It also says as a workaroud of setting the parameter AWS_STS_Endpoint , but did not work and for some people neither.

• Here a person say that it has a misconfig at chart level, but not sure if that could be my case, as I am not deploying from a custom helm chart but from helm upgrade --install fluent-bit fluent/fluent-bit --namespace fluent-bit so taking by default the values and tuning in some parameters for the serviceaccount and the configmap to be able to send logs to aws opensearch ...

• And here they ask for the necessary write permissions for the iam role, but that is not my case since I dont get permissions issues as the role is not assumed yet.

Just for the record this is my output opensearch plugin configuration:

   [OUTPUT]
        Name opensearch
        Match host.*
        Host vpc-xxxxr-eks-logs-test-qxxxxojgfi4d7fuoshm5e.eu-west-1.es.amazonaws.com
        Port 443
        AWS_Role_ARN arn:aws:iam::xxx:role/fluentbit-to-ope-xxxx-test-fluentbit-serviceaccount
        Logstash_Format On
        Logstash_Prefix node-logs
        Retry_Limit False
        AWS_Auth On
        AWS_Region eu-west-1
        tls On
        Trace_Output On
        Trace_Error On
        AWS_STS_Endpoint https://sts.eu-west-1.amazonaws.com

and that iam role (which is mapped from a k8s service account) has this policy attached

{
    "Statement": [
        {
            "Action": [
                "es:ESHttpPut",
                "es:ESHttpPost",
                "es:ESHttpGet",
                "es:ESHttpDelete"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": "es:*",
            "Effect": "Allow",
            "Resource": "arn:aws:es:eu-west-1:xxxxxx:domain/xxr-eks-logs-test"
        }
    ],
    "Version": "2012-10-17"
}

I think the problem is what mentioned at the beginning pod look for /var/run/secrets/eks.amazonaws.com/serviceaccount/aws-iam-token file but the env variable injected is AWS_WEB_IDENTITY_TOKEN_FILE : /var/run/secrets/eks.amazonaws.com/serviceaccount/token and that is why it does not find the token to assume the role.

How can I change the fluentbit configuration form the helm chart via parameters? It seems the injected service account token is the default managed by aws eks itself but the deployment pod from the helm chart look for a slightly diff path.

I will appreciate if someone can point me in a good direction 🙂