Setup to run Airflow in AWS ECS (Elastic Container Service) Fargate with autoscaling enabled for all services. All infrastructure is created with Cloudformation and Secrets are managed by AWS Secrets Manager.

• Create an AWS IAM User for the infrastructure deployment, with admin permissions
• Install AWS CLI running pip install awscli
• Install Docker
• Setup your IAM User credentials inside ~/.aws/config

    [profile my_aws_profile]
    aws_access_key_id = <my_access_key_id> 
    aws_secret_access_key = <my_secret_access_key>
    region = us-east-1

• Create a virtual environment
• Setup env variables in your .zshrc or .bashrc, or in your the terminal session that you are going to use:

	export AWS_REGION=us-east-1;
	export AWS_PROFILE=my_aws_profile;
	export ENVIRONMENT=dev;

make airflow-local

To deploy or update your stack run the following command:

make airflow-deploy

To rebuild Airflow Docker Image and push it to ECR (without infrastructure changes), run:

make airflow-push-image

To destroy your stack run the following command:

make airflow-destroy

After creating or updating a DAG you need to rebuild Airflow image, push it to ECR and then restart the airflow service. To do all that, you just need to execute:

make airflow-push-image

• Control all Airflow infrastructure from a single service.yml file.
• Metadata DB Passwords Managed with AWS Secrets Manager.
• Autoscaling enabled and configurable for all Airflow sub-services (workers, flower, webserver, scheduler)
• TODO: Continuous Integration using AWS CodePipeline
• TODO: Create isolated DAGs using docker_operator

  workers:
    port: 8793
    cpu: 1024
    memory: 2048
    desiredCount: 2
    autoscaling:
      maxCapacity: 8
      minCapacity: 2
      cpu:
        target: 70
        scaleInCooldown: 60
        scaleOutCooldown: 120
      memory:
        target: 70
        scaleInCooldown: 60
        scaleOutCooldown: 120

Look for AirflowWebServerEndpoint on outputs logged to your terminal.

    "cfn-airflow-webserver": [
        {
            "OutputKey": "AirflowWebServerEndpoint",
            "OutputValue": "airflow-dev-webserver-alb-1234567890.us-east-1.elb.amazonaws.com"
        }
    ],

Look for AirflowFlowerEndpoint on outputs logged to your terminal.

    "cfn-airflow-flower": [
        {
            "OutputKey": "AirflowFlowerEndpoint",
            "OutputValue": "airflow-dev-flower-alb-1234567890.us-east-1.elb.amazonaws.com"
        }
    ],

Inspired by the work done by Nicor88