Please refer to our User Guidance for information on how to use Analytical Platform Airflow.

Commit signature verification is enabled on this repository. To set this up follow the instructions on GitHub.

This project contains utility scripts under the scripts directory that support DAG development and validation.

These scripts import internal modules from the airflow/analytical_platform package. To ensure Python can resolve these imports, you must set the PYTHONPATH environment variable when running the scripts locally.

1. Install project dependencies using uv:

   uv venv
   
   uv pip install --requirements requirements.txt

2. Run scripts:

   # to validate the source workflow schema
   PYTHONPATH="airflow" uv run python scripts/workflow_schema_validation/main.py path/to/source/workflow.yml
   
   # to build the dist workflow.yml
   PYTHONPATH="airflow" uv run python scripts/workflow_generator/main.py path/to/source/workflow.yml
   
   # to validate the dist workflow.yml
   PYTHONPATH="airflow" uv run python scripts/workflow_validation/main.py path/to/dist/workflow.yml

The internal code is located under the airflow directory. Scripts use import statements like:

from analytical_platform.standard_operator import AnalyticalPlatformStandardOperator

Without setting PYTHONPATH=airflow, Python will not recognise analytical_platform as a valid module, since it's nested under the airflow directory.

1. Initialise submodules

   git submodule init
   
   git submodule update

2. Change into MWAA Local Runner directory

   cd aws-mwaa-local-runner

3. Build MWAA Local Runner

   [!NOTE] Depending on your internet speed, this may take a while

   ./mwaa-local-env build-image

4. Connect to Analytical Platform Compute

   aws-sso login
   
   aws-sso exec --profile "analytical-platform-compute-${ENVIRONMENT:-"development"}:platform-engineer-admin"
   
   aws eks update-kubeconfig --name "${AWS_SSO_PROFILE%%:*}"

5. Start MWAA Local Runner

   ./mwaa-local-env start

6. Log in to Airflow http://localhost:8080

• Username: admin
• Password: test