⚠️ this chart is the continuation of stable/airflow, see issue #6 for upgrade guide

Airflow is a platform to programmatically author, schedule, and monitor workflows.

helm repo add airflow-stable https://airflow-helm.github.io/charts
helm repo update

# Helm 3
helm install \
  [RELEASE_NAME] \
  airflow-stable/airflow \
  --version [VERSION] \
  --namespace [NAMESPACE] \
  --values ./custom-values.yaml

# Helm 2
helm install \
  airflow-stable/airflow \
  --name [RELEASE_NAME] \
  --version [VERSION] \
  --namespace [NAMESPACE] \
  --values ./custom-values.yaml

kubectl exec \
  -it \
  --namespace [NAMESPACE] \
  --container airflow-web \
  Deployment/[RELEASE_NAME]-web \
  /bin/bash

# then run commands like 
airflow create_user ...

Find chart version numbers under GitHub Releases:

• v7.14.X → v7.15.0
• v7.13.X → v7.14.0
• v7.12.X → v7.13.0
• v7.11.X → v7.12.0
• v7.10.X → v7.11.0
• v7.9.X → v7.10.0
• v7.8.X → v7.9.0
• v7.7.X → v7.8.0
• v7.6.X → v7.7.0
• v7.5.X → v7.6.0
• v7.4.X → v7.5.0
• v7.3.X → v7.4.0
• v7.2.X → v7.3.0
• v7.1.X → v7.2.0
• v7.0.X → v7.1.0
• v6.X.X → v7.0.0

Here are some starting points for your custom-values.yaml:

While we don't expose the airflow.cfg directly, you can use environment variables to set Airflow configs.

We expose the airflow.config value to make this easier:

airflow:
  config:
    ## Security
    AIRFLOW__CORE__SECURE_MODE: "True"
    AIRFLOW__API__AUTH_BACKEND: "airflow.api.auth.backend.deny_all"
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "False"
    AIRFLOW__WEBSERVER__RBAC: "False"

    ## DAGS
    AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: "30"
    AIRFLOW__CORE__LOAD_EXAMPLES: "False"

    ## Email (SMTP)
    AIRFLOW__EMAIL__EMAIL_BACKEND: "airflow.utils.email.send_email_smtp"
    AIRFLOW__SMTP__SMTP_HOST: "smtpmail.example.com"
    AIRFLOW__SMTP__SMTP_STARTTLS: "False"
    AIRFLOW__SMTP__SMTP_SSL: "False"
    AIRFLOW__SMTP__SMTP_PORT: "25"
    AIRFLOW__SMTP__SMTP_MAIL_FROM: "admin@example.com"

    ## Disable noisy "Handling signal: ttou" Gunicorn log messages
    GUNICORN_CMD_ARGS: "--log-level WARNING"

We expose the scheduler.connections value to specify Airflow Connections, which will be automatically imported by the airflow-scheduler when it starts up.

By default, we will delete and re-create connections each time the airflow-scheduler restarts. (If you want to manually modify a connection in the WebUI, you should disable this behaviour by setting scheduler.refreshConnections to false)

For example, to add a connection called my_aws:

scheduler:
  connections:
    - id: my_aws
      type: aws
      extra: |
        {
          "aws_access_key_id": "XXXXXXXX",
          "aws_secret_access_key": "XXXXXXXX",
          "region_name":"eu-central-1"
        }

If you don't want to store connections in your values.yaml, use scheduler.existingSecretConnections to specify the name of an existing Kubernetes Secret containing an add-connections.sh script. Note, your script will be run EACH TIME the airflow-scheduler Pod restarts, and scheduler.connections will not longer work.

Here is an example Secret you might create:

apiVersion: v1
kind: Secret
metadata:
  name: my-airflow-connections
type: Opaque
stringData:
  add-connections.sh: |
    #!/usr/bin/env bash

    # remove any existing connection
    airflow connections --delete \
      --conn_id "my_aws"
  
    # re-add your custom connection
    airflow connections --add \
      --conn_id "my_aws" \
      --conn_type "aws" \
      --conn_extra "{\"aws_access_key_id\": \"XXXXXXXX\", \"aws_secret_access_key\": \"XXXXXXXX\", \"region_name\":\"eu-central-1\"}"

We expose the scheduler.variables value to specify Airflow Variables, which will be automatically imported by the airflow-scheduler when it starts up.

For example, to specify a variable called environment:

scheduler:
  variables: |
    { "environment": "dev" }

We expose the scheduler.pools value to specify Airflow Pools, which will be automatically imported by the Airflow scheduler when it starts up.

For example, to create a pool called example:

scheduler:
  pools: |
    {
      "example": {
        "description": "This is an example pool with 2 slots.",
        "slots": 2
      }
    }

We expose the airflow.extraEnv value to mount extra environment variables, this can be used to pass sensitive configs to Airflow.

For example, passing a Fernet key and LDAP password, (the airflow and ldap Kubernetes Secrets must already exist):

airflow:
  extraEnv:
    - name: AIRFLOW__CORE__FERNET_KEY
      valueFrom:
        secretKeyRef:
          name: airflow
          key: fernet-key
    - name: AIRFLOW__LDAP__BIND_PASSWORD
      valueFrom:
        secretKeyRef:
          name: ldap
          key: password

We expose the airflow.extraConfigmapMounts value to mount extra Kubernetes ConfigMaps.

For example, a webserver_config.py file:

airflow:
  extraConfigmapMounts:
    - name: my-webserver-config
      mountPath: /opt/airflow/webserver_config.py
      configMap: my-airflow-webserver-config
      readOnly: true
      subPath: webserver_config.py

To create the my-airflow-webserver-config ConfigMap, you could use:

kubectl create configmap \
  my-airflow-webserver-config \
  --from-file=webserver_config.py \
  --namespace airflow

We expose the airflow.extraPipPackages and web.extraPipPackages values to install Python Pip packages, these will work with any pip package that you can install with pip install XXXX.

For example, enabling the airflow airflow-exporter package:

airflow:
  extraPipPackages:
    - "airflow-exporter==1.3.1"

For example, you may be using flask_oauthlib to integrate with Okta/Google/etc for authorizing WebUI users:

web:
  extraPipPackages:
    - "apache-airflow[google_auth]==1.10.12"

This chart provides an optional Kubernetes Ingress resource, for accessing airflow-webserver and airflow-flower outside of the cluster.

If you already have something hosted at the root of your domain, you might want to place airflow under a URL-prefix:

• http://example.com/airflow/
• http://example.com/airflow/flower

In this example, would set these values:

web:
  baseUrl: "http://example.com/airflow/"

flower:
  urlPrefix: "/airflow/flower"

ingress:
  web:
    path: "/airflow"

  flower:
    path: "/airflow/flower"

We expose the ingress.web.precedingPaths and ingress.web.succeedingPaths values, which are before and after the default path respectively.

A common use-case is enabling https with the aws-alb-ingress-controller ssl-redirect, which needs a redirect path to be hit before the airflow-webserver one.

You would set the values of precedingPaths as the following:

ingress:
  web:
    precedingPaths:
      - path: "/*"
        serviceName: "ssl-redirect"
        servicePort: "use-annotation"

We use a Kubernetes StatefulSet for the Celery workers, this allows the webserver to requests logs from each workers individually, with a fixed DNS name.

Celery workers can be scaled using the Horizontal Pod Autoscaler. To enable autoscaling, you must set workers.autoscaling.enabled=true, then provide workers.autoscaling.maxReplicas, and workers.replicas for the minimum amount.

Assume every task a worker executes consumes approximately 200Mi memory, that means memory is a good metric for utilisation monitoring. For a worker pod you can calculate it: WORKER_CONCURRENCY * 200Mi, so for 10 tasks a worker will consume ~2Gi of memory.

In the following config if a worker consumes 80% of 2Gi (which will happen if it runs 9-10 tasks at the same time), an autoscaling event will be triggered, and a new worker will be added. If you have many tasks in a queue, Kubernetes will keep adding workers until maxReplicas reached, in this case 16.

workers:
  # the initial/minimum number of workers
  replicas: 2

  resources:
    requests:
      memory: "2Gi"

  podDisruptionBudget:
    enabled: true
    ## prevents losing more than 20% of current worker task slots in a voluntary disruption
    maxUnavailable: "20%"

  autoscaling:
    enabled: true
    maxReplicas: 16
    metrics:
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

  celery:
    instances: 10

    ## wait at most 9min for running tasks to complete before SIGTERM
    ## WARNING: 
    ##  - some cluster-autoscaler (GKE) will not respect graceful 
    ##    termination periods over 10min
    gracefullTermination: true
    gracefullTerminationPeriod: 540

  ## how many seconds (after the 9min) to wait before SIGKILL
  terminationPeriod: 60

dags:
  git:
    gitSync:
      resources:
        requests:
          ## IMPORTANT! for autoscaling to work
          memory: "64Mi"

We expose the workers.secrets value to allow mounting secrets at {workers.secretsDir}/<secret-name> in airflow-worker Pods.

For example, mounting password Secrets:

workers:
  secretsDir: /var/airflow/secrets
  secrets:
    - redshift-user
    - redshift-password
    - elasticsearch-user
    - elasticsearch-password

With the above configuration, you could read the redshift-user password from within a DAG or Python function using:

import os
from pathlib import Path

def get_secret(secret_name):
    secrets_dir = Path('/var/airflow/secrets')
    secret_path = secrets_dir / secret_name
    assert secret_path.exists(), f'could not find {secret_name} at {secret_path}'
    secret_data = secret_path.read_text().strip()
    return secret_data

redshift_user = get_secret('redshift-user')

To create the redshift-user Secret, you could use:

kubectl create secret generic \
  redshift-user \
  --from-literal=redshift-user=MY_REDSHIFT_USERNAME \
  --namespace airflow

We expose the extraManifests.[] value to add custom Kubernetes manifests to the chart.

For example, adding a BackendConfig resource for GKE:

extraManifests:
  - apiVersion: cloud.google.com/v1beta1
    kind: BackendConfig
    metadata:
      name: "{{ .Release.Name }}-test"
    spec:
      securityPolicy:
        name: "gcp-cloud-armor-policy-test"

If the value scheduler.initdb is set to true (this is the default), the airflow-scheduler container will run airflow upgradedb || airflow db upgrade as part of its startup script.

If the value scheduler.preinitdb is set to true, then we ALSO RUN airflow upgradedb || airflow db upgrade in an init-container (retrying 5 times). This is unusually NOT necessary unless your synced DAGs include custom database hooks that prevent airflow upgradedb || airflow db upgrade from running.

PostgreSQL is the default database in this chart, because we use insecure username/password combinations by default, you should create secure credentials before installing the Helm chart.

Example bash command to create the required Kubernetes Secrets:

# set postgress password
kubectl create secret generic \
  airflow-postgresql \
  --from-literal=postgresql-password=$(openssl rand -base64 13) \
  --namespace airflow

# set redis password
kubectl create secret generic \
  airflow-redis \
  --from-literal=redis-password=$(openssl rand -base64 13) \
  --namespace airflow

Example values.yaml, to use those secrets:

postgresql:
  existingSecret: airflow-postgresql

redis:
  existingSecret: airflow-redis

🛑️️ the embedded PostgreSQL is NOT SUITABLE for production, you should configure one of the following external databases

Example values for an external Postgres database, with an existing airflow_cluster1 database:

externalDatabase:
  type: postgres
  host: postgres.example.org
  port: 5432
  database: airflow_cluster1
  user: airflow_cluster1
  passwordSecret: "airflow-cluster1-postgres-password"
  passwordSecretKey: "postgresql-password"

⚠️ you must set explicit_defaults_for_timestamp=1 in your MySQL instance, see here

Example values for an external MySQL database, with an existing airflow_cluster1 database:

externalDatabase:
  type: mysql
  host: mysql.example.org
  port: 3306
  database: airflow_cluster1
  user: airflow_cluster1
  passwordSecret: "airflow-cluster1-mysql-password"
  passwordSecretKey: "mysql-password"

🛑️️ you should persist logs in a production deployment using one of the following methods

By default, logs from the airflow-web/scheduler/worker are written within the Docker container's filesystem, therefore any restart of the pod will wipe the logs.

You must give airflow credentials for it to read/write on the remote bucket, this can be achieved with AIRFLOW__CORE__REMOTE_LOG_CONN_ID, or by using something like Workload Identity (GKE), or IAM Roles for Service Accounts (EKS).

Example, using AIRFLOW__CORE__REMOTE_LOG_CONN_ID (can be used with AWS too):

airflow:
  config:
    AIRFLOW__CORE__REMOTE_LOGGING: "True"
    AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER: "gs://<<MY-BUCKET-NAME>>/airflow/logs"
    AIRFLOW__CORE__REMOTE_LOG_CONN_ID: "google_cloud_airflow"

scheduler:
  connections:
    - id: google_cloud_airflow
      type: google_cloud_platform
      extra: |-
        {
         "extra__google_cloud_platform__num_retries": "5",
         "extra__google_cloud_platform__keyfile_dict": "{...}"
        }

Example, using Workload Identity (GKE):

airflow:
  config:
    AIRFLOW__CORE__REMOTE_LOGGING: "True"
    AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER: "gs://<<MY-BUCKET-NAME>>/airflow/logs"
    AIRFLOW__CORE__REMOTE_LOG_CONN_ID: "google_cloud_default"

serviceAccount:
  annotations:
    iam.gke.io/gcp-service-account: "<<MY-ROLE-NAME>>@<<MY-PROJECT-NAME>>.iam.gserviceaccount.com"

Example, using IAM Roles for Service Accounts (EKS):

airflow:
  config:
    AIRFLOW__CORE__REMOTE_LOGGING: "True"
    AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER: "s3://<<MY-BUCKET-NAME>>/airflow/logs"
    AIRFLOW__CORE__REMOTE_LOG_CONN_ID: "aws_default"

scheduler:
  securityContext:
    fsGroup: 65534

web:
  securityContext:
    fsGroup: 65534

workers:
  securityContext:
    fsGroup: 65534

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::XXXXXXXXXX:role/<<MY-ROLE-NAME>>"

logs:
  persistence:
    enabled: true

A ServiceMonitor is a resource introduced by the Prometheus Operator.

To be able to expose Airflow metrics to Prometheus you will need install a plugin, one option is epoch8/airflow-exporter, which exposes dag and task based metrics from Airflow.

For more information, see the serviceMonitor section of values.yaml.

⚠️ specifying known_hosts inside dags.git.secret reduces the possibility of a man-in-the-middle attack, however, if you want to implicitly trust all repo host signatures set dags.git.sshKeyscan to true

This method places a git sidecar in each worker/scheduler/web Pod, that syncs your git repo into the dag folder every dags.git.gitSync.refreshTime seconds.

dags:
  git:
    #ssh://git@example.com:22/REPOSITORY.git
    url: git@github.com:USERNAME/REPOSITORY.git
    ref: master
    secret: airflow-git-keys
    privateKeyName: id_rsa
    repoHost: github.com
    repoPort: 22

    gitSync:
      enabled: true
      refreshTime: 60

You can create the dags.git.secret from your local $HOME/.ssh folder using:

kubectl create secret generic \
  airflow-git-keys \
  --from-file=id_rsa=$HOME/.ssh/id_rsa \
  --from-file=id_rsa.pub=$HOME/.ssh/id_rsa.pub \
  --from-file=known_hosts=$HOME/.ssh/known_hosts \
  --namespace airflow

⚠️️ this method requires a PersistentVolumeClaim which supports accessMode = ReadOnlyMany or ReadWriteMany

This method stores your DAGs in a PersistentVolume.

You must configure some external system to ensure this volume has your latest DAGs. For example, you could use your CI/CD pipeline system to preform a sync as changes are pushed to a git repo.

Since ALL Pods MUST HAVE the same collection of DAG files, it is recommended to create just one PVC that is shared. To share a PVC with multiple Pods, the PVC needs to have accessMode set to ReadOnlyMany or ReadWriteMany (Note: different StorageClass support different access modes).

If you are using Kubernetes on a public cloud, a persistent volume controller is likely built in:

• Amazon EKS
• Azure AKS
• Google GKE

For example, to use the storage class called default:

dags:
  persistence:
    enabled: true
    storageClass: default
    accessMode: ReadOnlyMany
    size: 1Gi

⚠️ this method requires a PersistentVolumeClaim which supports accessMode = ReadWriteMany

This method stores both DAGs and logs on the same PersistentVolume.

Here's an approach that achieves this:

• Configure airflow.extraVolume and airflow.extraVolumeMount to put a volume at /opt/airflow/efs
• Configure dags.persistence.enabled and logs.persistence.enabled to be false
• Configure dags.path to be /opt/airflow/efs/dags
• Configure logs.path to be /opt/airflow/efs/logs

This method stores your DAGs inside the container image.

This chart uses the official apache/airflow image, extend this image and COPY your DAGs into the dags.path folder:

FROM apache/airflow:1.10.12-python3.6

# NOTE: dag path is set with the `dags.path` value
COPY ./my_dag_folder /opt/airflow/dags

The following values can be used to specify the container image:

airflow:
  image:
    repository: MY_REPO
    tag: MY_TAG

⚠️ if you update the requirements.txt, you will have to restart each worker Pod for changes to take effect, you might consider using airflow.extraPipPackages instead

We expose the dags.installRequirements value to pip install any requirements.txt found at the root of your dags.path folder as airflow-worker Pods start.

Full documentation can be found in the comments of the values.yaml file, but a high level overview is provided here.

Global Values:

Airflow Scheduler values:

Airflow Webserver Values:

Airflow Worker Values:

Airflow Flower Values:

Airflow Logs Values:

Airflow DAGs Values:

Airflow Ingress Values:

Airflow Kubernetes Values:

Airflow Database (Internal PostgreSQL) Values:

Airflow Database (External) Values:

Airflow Redis (Internal) Values:

Airflow Redis (External) Values:

Airflow Prometheus Values: